# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=525 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 526 --- ## [My grokstash pattern matches only the first match and skips the rest into the message body](https://discuss.elastic.co/t/my-grokstash-pattern-matches-only-the-first-match-and-skips-the-rest-into-the-message-body/186518)
**Author:** [@kganesan](https://discuss.elastic.co/u/kganesan)\ **Replies:** 3\ **Last updated:** [June 19, 2019, 6:19pm UTC](https://discuss.elastic.co/t/my-grokstash-pattern-matches-only-the-first-match-and-skips-the-rest-into-the-message-body/186518 "2019-06-19T18:19:53Z")
I am using logtsash grok filters for extracting useful information from the log files. I specifically need to extract recurring stopwatch lines from the log file info. My grok filter does match the first set of stopwat… --- ## [Logstash Logging to /var/log/messages](https://discuss.elastic.co/t/logstash-logging-to-var-log-messages/186085)
**Author:** [@DougR](https://discuss.elastic.co/u/DougR)\ **Replies:** 2\ **Last updated:** [June 19, 2019, 4:52pm UTC](https://discuss.elastic.co/t/logstash-logging-to-var-log-messages/186085 "2019-06-19T16:52:23Z")
Logstash is logging a duplicate of all data being logged to /var/log/logstash/logstash-plain.log to /var/log/messages. It appears that many others have experienced the same issue, and that the culprit is often the follow… --- ## [If/else within Logstash output plugin](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965)
**Author:** [@g.le](https://discuss.elastic.co/u/g.le)\ **Replies:** 13\ **Last updated:** [June 19, 2019, 4:33pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965 "2019-06-19T16:33:46Z")
Hello, I'm having a Logstash configuration similar to the below: input { beats { port =\> 5044 } } filter { clone { clones =\> \["local-dc"\] add\_tag =\> \["cloned"\] } } output { if "cloned" in \[tags\] {… --- ## [Logstash update if id matches](https://discuss.elastic.co/t/logstash-update-if-id-matches/186472)
**Author:** [@Aymen\_Ben\_Moussa](https://discuss.elastic.co/u/Aymen_Ben_Moussa)\ **Replies:** 5\ **Last updated:** [June 19, 2019, 2:42pm UTC](https://discuss.elastic.co/t/logstash-update-if-id-matches/186472 "2019-06-19T14:42:40Z")
is there a possibility to update a document if id matches else insert ? Alright i'm reading a kafka topic and this a sample of my topic \` \> "schema":{"type":"struct","fields":\[{"type":"string","optional":true,"field":… --- ## [Installing logstash plugin inside docker container](https://discuss.elastic.co/t/installing-logstash-plugin-inside-docker-container/186477)
**Author:** [@thomas.heuberger](https://discuss.elastic.co/u/thomas.heuberger)\ **Replies:** 4\ **Last updated:** [June 19, 2019, 2:05pm UTC](https://discuss.elastic.co/t/installing-logstash-plugin-inside-docker-container/186477 "2019-06-19T14:05:42Z")
So, when trying to install a logstash plugin inside a docker container (using an interactive shell, or a Dockerfile or docker-compose , doesn't matter), this happens: sh-4.2$ ./logstash-plugin install logstash-codec-sfl… --- ## [How to apply different filters based on particular name of files without using Filebeat](https://discuss.elastic.co/t/how-to-apply-different-filters-based-on-particular-name-of-files-without-using-filebeat/186272)
**Author:** [@dundundundun](https://discuss.elastic.co/u/dundundundun)\ **Replies:** 4\ **Last updated:** [June 18, 2019, 7:00pm UTC](https://discuss.elastic.co/t/how-to-apply-different-filters-based-on-particular-name-of-files-without-using-filebeat/186272 "2019-06-18T19:00:19Z")
I want to perform a unique grok filter per log. So let's say for 1.log I want to perform filter A, 2.log I want filter B, and so on. I've copied my input section in the logstash configuration file with psuedo code in the… --- ## [Logstash is replacing : with =\> How can I get : on the output](https://discuss.elastic.co/t/logstash-is-replacing-with-how-can-i-get-on-the-output/186433)
**Author:** [@Raj\_Sekhar](https://discuss.elastic.co/u/Raj_Sekhar)\ **Replies:** 7\ **Last updated:** [June 19, 2019, 1:15pm UTC](https://discuss.elastic.co/t/logstash-is-replacing-with-how-can-i-get-on-the-output/186433 "2019-06-19T13:15:56Z")
Logstash is replacing : with =\> How can I get : on the output My input file: { "customfield\_10150": \[ {"key": "caanyimi", "displayName": "Anyimi, Charles"} \] } Output: "name""=\>""caanyimi"", ""emailAddress""=\>"… --- ## [Create several Index from same filebeats source](https://discuss.elastic.co/t/create-several-index-from-same-filebeats-source/186115)
**Author:** [@robymemo](https://discuss.elastic.co/u/robymemo)\ **Replies:** 2\ **Last updated:** [June 19, 2019, 12:50pm UTC](https://discuss.elastic.co/t/create-several-index-from-same-filebeats-source/186115 "2019-06-19T12:50:11Z")
Hi to all, i'm newbi on this tecnology. I'm trying to generate 2 different index basing on the same input files from beats. My basic configuration (1) (working) create the index 'log\_reporti\_YYYYMM' taking all the '.l… --- ## [Logstash ingested CSV file twice](https://discuss.elastic.co/t/logstash-ingested-csv-file-twice/186351)
**Author:** [@victoravr](https://discuss.elastic.co/u/victoravr)\ **Replies:** 4\ **Last updated:** [June 19, 2019, 12:47pm UTC](https://discuss.elastic.co/t/logstash-ingested-csv-file-twice/186351 "2019-06-19T12:47:58Z")
For some reason Logstash processed CSV file twice. Could you please help to find out why? Running logstash.version 6.5.3. input { file { path =\> "/path/to/\*.csv" start\_position =\> "beginning" mode =\> "read" file\_c… --- ## [Best way to Input or Output in Logstash](https://discuss.elastic.co/t/best-way-to-input-or-output-in-logstash/186464)
**Author:** [@gabrielesteveslima](https://discuss.elastic.co/u/gabrielesteveslima)\ **Replies:** 0\ **Last updated:** [June 19, 2019, 12:19pm UTC](https://discuss.elastic.co/t/best-way-to-input-or-output-in-logstash/186464 "2019-06-19T12:19:22Z")
I am involved in a project that has the following characteristics: 1 - In the current scenario we have several third-party systems, which record differently in Oracle database that we have ownership, that is, we have do… --- ## [\[Logstash\]\[Filebeat\] multiple logstash pipeline with filebeat](https://discuss.elastic.co/t/logstash-filebeat-multiple-logstash-pipeline-with-filebeat/186203)
**Author:** [@patimothe](https://discuss.elastic.co/u/patimothe)\ **Replies:** 2\ **Last updated:** [June 19, 2019, 12:13pm UTC](https://discuss.elastic.co/t/logstash-filebeat-multiple-logstash-pipeline-with-filebeat/186203 "2019-06-19T12:13:49Z")
Hello, I installed on a Server logstash, elasticsearch and kibana. I've got different pipelines on logstash to parse different type of logs: example: "TPW-pipeline.conf", "weblogic-pipeline.conf", "batch-pipeline.conf… --- ## [Stuck at "Successfully started Lgstash API endpoint {:port=\>9600}" and nothing happend when import csv](https://discuss.elastic.co/t/stuck-at-successfully-started-lgstash-api-endpoint-port-9600-and-nothing-happend-when-import-csv/179462)
**Author:** [@Sean\_Donghyun\_Park](https://discuss.elastic.co/u/Sean_Donghyun_Park)\ **Replies:** 2\ **Last updated:** [May 22, 2019, 10:35am UTC](https://discuss.elastic.co/t/stuck-at-successfully-started-lgstash-api-endpoint-port-9600-and-nothing-happend-when-import-csv/179462 "2019-05-22T10:35:41Z")
Hi. I got problem when I import csv file to logstash and output to elasticsearch. id,title,defaultLanguage,caption,start,duration 1,Triple Frontier Trailer #2 (2019) | Movieclips Trailers,en,I have never had a feeling a… --- ## [Is it possible to collect data from my website with logstash](https://discuss.elastic.co/t/is-it-possible-to-collect-data-from-my-website-with-logstash/185395)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 6\ **Last updated:** [June 19, 2019, 9:29am UTC](https://discuss.elastic.co/t/is-it-possible-to-collect-data-from-my-website-with-logstash/185395 "2019-06-19T09:29:14Z")
hi dears, i have a web site for selling products and i want to know if it possible to collect data from a website with logstash, if there is any example Thank you very much --- ## [Variable string empty in ruby filter](https://discuss.elastic.co/t/variable-string-empty-in-ruby-filter/186367)
**Author:** [@manunc](https://discuss.elastic.co/u/manunc)\ **Replies:** 2\ **Last updated:** [June 19, 2019, 8:28am UTC](https://discuss.elastic.co/t/variable-string-empty-in-ruby-filter/186367 "2019-06-19T08:28:27Z")
Hello, I am using ELK 5.6 (for several reasons) I dont not succeed to assign a variable string using this ruby code: s = event.get("\[sigfox\]\[data\]") tab = s.split(/(\\w{2})(\\w{2})(\\w{20})/) pack\_id … --- ## [Roles / user with puppet](https://discuss.elastic.co/t/roles-user-with-puppet/186400)
**Author:** [@jof300](https://discuss.elastic.co/u/jof300)\ **Replies:** 0\ **Last updated:** [June 19, 2019, 7:35am UTC](https://discuss.elastic.co/t/roles-user-with-puppet/186400 "2019-06-19T07:35:42Z")
Hello, I am setting up TLS for my new cluster ELK (7.1.0). Everything is working well, except the roles / users management. I am using Puppet to create roles and users. The strange things are : I create 2 users with… --- ## [You are using a deprecated config setting "document\_type" set in elasticsearch](https://discuss.elastic.co/t/you-are-using-a-deprecated-config-setting-document-type-set-in-elasticsearch/184826)
**Author:** [@madcon](https://discuss.elastic.co/u/madcon)\ **Replies:** 3\ **Last updated:** [June 19, 2019, 5:50am UTC](https://discuss.elastic.co/t/you-are-using-a-deprecated-config-setting-document-type-set-in-elasticsearch/184826 "2019-06-19T05:50:54Z")
Hello All, Fairly new to ELK stack I did encountered the following warning below during my setup. /\[WARN \]\[logstash.outputs.elasticsearch\] You are using a deprecated config setting "document\_type" set in elasticsearch.… --- ## [Is there any way to access all cloudwatch log groups without providing the exact name?](https://discuss.elastic.co/t/is-there-any-way-to-access-all-cloudwatch-log-groups-without-providing-the-exact-name/186349)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 4\ **Last updated:** [June 19, 2019, 12:21am UTC](https://discuss.elastic.co/t/is-there-any-way-to-access-all-cloudwatch-log-groups-without-providing-the-exact-name/186349 "2019-06-19T00:21:18Z")
Hi everyone, question about Logstash input plugin for Cloudwatch logs. I want to get all the logs in the log groups that start with /aws/lambda/, is that possible? I tried the below configuration with log\_group =\> \["/a… --- ## ["delimiter" field behaving weirdly with \\n char for the line codec with UTF-16](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341)
**Author:** [@res260](https://discuss.elastic.co/u/res260)\ **Replies:** 12\ **Last updated:** [June 18, 2019, 9:21pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341 "2019-06-18T21:21:20Z")
Hi, I’m trying to parse UTF-16-encoded (with BOM), CRLF newlines CSV data from a TCP input, but I cant get it to be parsed correctly. In my .conf file: input { tcp { port =\> 5008 type =\> "csv" codec =\> line { … --- ## [Output file plugin not creating timestamped file](https://discuss.elastic.co/t/output-file-plugin-not-creating-timestamped-file/186334)
**Author:** [@erikmathis](https://discuss.elastic.co/u/erikmathis)\ **Replies:** 2\ **Last updated:** [June 18, 2019, 8:30pm UTC](https://discuss.elastic.co/t/output-file-plugin-not-creating-timestamped-file/186334 "2019-06-18T20:30:40Z")
output { file{ path =\> "/var/log/kube-audit/audit-%{+YYYY.MM.dd.HH}.log" create\_if\_deleted =\> true id =\> "file" } The file looks ends up being /var/log/kube-audit/audit-.log. The timestamp works fine in th… --- ## [Enabling SSL on Logstash](https://discuss.elastic.co/t/enabling-ssl-on-logstash/186309)
**Author:** [@bkasrai](https://discuss.elastic.co/u/bkasrai)\ **Replies:** 2\ **Last updated:** [June 18, 2019, 8:11pm UTC](https://discuss.elastic.co/t/enabling-ssl-on-logstash/186309 "2019-06-18T20:11:18Z")
Good afternoon, Over the past few days, i recently enabled SSO\\SAML authentication on my staging environment. By doing so, Logstash will no longer send data over to ES given the ES yml updates with the addition of the h… --- ## [Logstash Filtering Grabbing Partial Date](https://discuss.elastic.co/t/logstash-filtering-grabbing-partial-date/186257)
**Author:** [@Elk\_huh](https://discuss.elastic.co/u/Elk_huh)\ **Replies:** 1\ **Last updated:** [June 18, 2019, 1:44pm UTC](https://discuss.elastic.co/t/logstash-filtering-grabbing-partial-date/186257 "2019-06-18T13:44:18Z")
date { match =\> \["event\_at", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"\] } How would i Grab just "yyyy-MM-dd'T'HH" and send it to a new field ? --- ## [Incorrect mapping not matching grok filers and fields .conf file](https://discuss.elastic.co/t/incorrect-mapping-not-matching-grok-filers-and-fields-conf-file/186288)
**Author:** [@asad\_ali](https://discuss.elastic.co/u/asad_ali)\ **Replies:** 4\ **Last updated:** [June 18, 2019, 6:28pm UTC](https://discuss.elastic.co/t/incorrect-mapping-not-matching-grok-filers-and-fields-conf-file/186288 "2019-06-18T18:28:30Z")
Hey folks, I have .conf as input { beats { port =\> 5044} } filter { grok { match =\> \[ "message", "%{TIMESTAMP\_ISO8601:timestamp\_sting}%{SPACE}%{GREEDYDATA:line}" \] } date { match =\> \["timestamp… --- ## [Choose filter-kv or another from list for such type logs](https://discuss.elastic.co/t/choose-filter-kv-or-another-from-list-for-such-type-logs/186324)
**Author:** [@freelanseod](https://discuss.elastic.co/u/freelanseod)\ **Replies:** 1\ **Last updated:** [June 18, 2019, 6:26pm UTC](https://discuss.elastic.co/t/choose-filter-kv-or-another-from-list-for-such-type-logs/186324 "2019-06-18T18:26:45Z")
Hello to everyone I have logs at docker-compose such like this 2019-06-18 16:26:03.639 INFO \[kafka-producer-network-thread | testSrvProducer-1\] \[kafka.producer.impl.MessageProducerKafkaImpl\] \[\] - delivered to topic=t… --- ## [Logstash Aggregate Plugin: Unify two different task\_id](https://discuss.elastic.co/t/logstash-aggregate-plugin-unify-two-different-task-id/185712)
**Author:** [@mpappatella](https://discuss.elastic.co/u/mpappatella)\ **Replies:** 2\ **Last updated:** [June 18, 2019, 6:14pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-unify-two-different-task-id/185712 "2019-06-18T18:14:43Z")
Hello i'm new here, I am looking to unify in a same line, 4 different windows events received by beats. Specifically 4 events that ADFS generates when someone authenticates. I need to unify the Activity ID and the Inst… --- ## [Unable to add new fields in the output event](https://discuss.elastic.co/t/unable-to-add-new-fields-in-the-output-event/185905)
**Author:** [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\ **Replies:** 2\ **Last updated:** [June 18, 2019, 4:52pm UTC](https://discuss.elastic.co/t/unable-to-add-new-fields-in-the-output-event/185905 "2019-06-18T16:52:54Z")
I am trying to add a new field with the name "failed\_reason\_new". The following filter is not working. I am getting the below error message. C:/Demo/logstash-7.1.1/logstash-core/lib/logstash/compiler.rb:49:in \`compile… --- ## [2 CSV with relationship](https://discuss.elastic.co/t/2-csv-with-relationship/186308)
**Author:** [@rori1](https://discuss.elastic.co/u/rori1)\ **Replies:** 1\ **Last updated:** [June 18, 2019, 4:22pm UTC](https://discuss.elastic.co/t/2-csv-with-relationship/186308 "2019-06-18T16:22:01Z")
Hello, I have 2 CSV files: 1 containing a list of countries and the second the list of cities of countries. These two files have a CountryId column that allows you to do the relationship. After integrating the list of … --- ## [Error sending bulk request do Elasticsearch](https://discuss.elastic.co/t/error-sending-bulk-request-do-elasticsearch/183193)
**Author:** [@luisdiasbh](https://discuss.elastic.co/u/luisdiasbh)\ **Replies:** 3\ **Last updated:** [June 18, 2019, 2:54pm UTC](https://discuss.elastic.co/t/error-sending-bulk-request-do-elasticsearch/183193 "2019-06-18T14:54:18Z")
I'm able to read some data from hive table throught jdbc connection but when it will be inserted on elastic i have the following error: \[ERROR\] 2019-05-28 17:39:36.643 \[\[main\]\>worker1\] elasticsearch - An unknown error … --- ## [Cant run logstash using logstash -f](https://discuss.elastic.co/t/cant-run-logstash-using-logstash-f/186284)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 1\ **Last updated:** [June 18, 2019, 2:53pm UTC](https://discuss.elastic.co/t/cant-run-logstash-using-logstash-f/186284 "2019-06-18T14:53:05Z")
whe i try to run logstash in order to collect data using the commande "logstash -f" i obtain the following message and logstash is shutdown: 2019-06-18 16:33:44.152 \[LogStash::Runner\] runner - Starting Logstash {"logsta… --- ## [Jdbc streaming renaming fields](https://discuss.elastic.co/t/jdbc-streaming-renaming-fields/186263)
**Author:** [@jonathan01](https://discuss.elastic.co/u/jonathan01)\ **Replies:** 2\ **Last updated:** [June 18, 2019, 2:04pm UTC](https://discuss.elastic.co/t/jdbc-streaming-renaming-fields/186263 "2019-06-18T14:04:40Z")
I am using the jdbc streaming plugin and want to take the results and put them into new fields. Below is the code that I am using jdbc\_streaming { jdbc\_driver\_library =\> "C:\\logstash-7.0.0\\bin\\sqljdbc42.jar" jdbc\_dri… --- ## [Http\_poller not able to handle 302](https://discuss.elastic.co/t/http-poller-not-able-to-handle-302/186239)
**Author:** [@sjabiulla](https://discuss.elastic.co/u/sjabiulla)\ **Replies:** 1\ **Last updated:** [June 18, 2019, 1:47pm UTC](https://discuss.elastic.co/t/http-poller-not-able-to-handle-302/186239 "2019-06-18T13:47:55Z")
I have an url ( lets say URL1, this is just http ) which just redirects to another url ( lets say URL2, this is https ) on invoking. When I try invoking the URL1 via curl , it clearly gives me 302 . But when I do this vi… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=524) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=526)