# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=114 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 115 --- ## [Environment variable in if condition is not working](https://discuss.elastic.co/t/environment-variable-in-if-condition-is-not-working/314863)
**Author:** [@seb.sch](https://discuss.elastic.co/u/seb.sch)\ **Replies:** 4\ **Last updated:** [September 21, 2022, 4:18pm UTC](https://discuss.elastic.co/t/environment-variable-in-if-condition-is-not-working/314863 "2022-09-21T16:18:36Z")
Hello there, I have the following pipeline output configuration: output { if ("${ELASTICSEARCH\_HOSTS:}" == "http://elasticsearch:9200") { elasticsearch { action =\> "index" index =\> "street" host… --- ## [How to set a schedule on Filter or reuse the content of a event](https://discuss.elastic.co/t/how-to-set-a-schedule-on-filter-or-reuse-the-content-of-a-event/314520)
**Author:** [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)\ **Replies:** 4\ **Last updated:** [September 21, 2022, 3:49pm UTC](https://discuss.elastic.co/t/how-to-set-a-schedule-on-filter-or-reuse-the-content-of-a-event/314520 "2022-09-21T15:49:42Z")
Hello everyone, need a share of your help again. This time with Logstash. The context of the problem is: I got a pipeline which goes into an API to retrieve a token with a POST, once I got this token I use it on anothe… --- ## [Pipelines.yml ignored on logstash](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807)
**Author:** [@reswob](https://discuss.elastic.co/u/reswob)\ **Replies:** 9\ **Last updated:** [September 21, 2022, 1:17pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807 "2022-09-21T13:17:28Z")
I have looked at the following: I have LS 8.3.3 and here is version 1 of pipelines.yml located in /opt/logstash/config/pipelines.yml - pipeline.id: main path.config: "/opt/logstash/lab\_conf/\*.conf" - pipeline.id:… --- ## [DB connection information](https://discuss.elastic.co/t/db-connection-information/314793)
**Author:** [@KiranMohan](https://discuss.elastic.co/u/KiranMohan)\ **Replies:** 0\ **Last updated:** [September 20, 2022, 4:39pm UTC](https://discuss.elastic.co/t/db-connection-information/314793 "2022-09-20T16:39:35Z")
It seems Logstash (7.10) Input JDBC plugin is leaking DB connections in one of the production servers. Is it possible to trace the DB connection information? The underlying Sequel library has an option to log\_connectio… --- ## [Logstash-extract multiple subfield values in multiple events](https://discuss.elastic.co/t/logstash-extract-multiple-subfield-values-in-multiple-events/314682)
**Author:** [@Akshay\_Kulkarni](https://discuss.elastic.co/u/Akshay_Kulkarni)\ **Replies:** 6\ **Last updated:** [September 21, 2022, 5:27am UTC](https://discuss.elastic.co/t/logstash-extract-multiple-subfield-values-in-multiple-events/314682 "2022-09-21T05:27:33Z")
Hi, I have below type of events(fields value can be dynamic). I'm trying to split field's key, value as new event. I'm able to do it for two fields(TOTAl\_VOLUME, SUCCESS\_VOLUME), but when i try for 3rd field, logstash … --- ## [How to convert this date type to a readable type '1661126482845'](https://discuss.elastic.co/t/how-to-convert-this-date-type-to-a-readable-type-1661126482845/314782)
**Author:** [@Vitoria\_De\_Lara](https://discuss.elastic.co/u/Vitoria_De_Lara)\ **Replies:** 1\ **Last updated:** [September 20, 2022, 4:00pm UTC](https://discuss.elastic.co/t/how-to-convert-this-date-type-to-a-readable-type-1661126482845/314782 "2022-09-20T16:00:36Z")
I need to convert this date type to a readable date type in my file. conf, I'm using the json filter. I've tried several conversion models but it's not working. can anybody help me? --- ## [I m unable to restart the logstash after updating the conf.d file](https://discuss.elastic.co/t/i-m-unable-to-restart-the-logstash-after-updating-the-conf-d-file/314768)
**Author:** [@prashanthk](https://discuss.elastic.co/u/prashanthk)\ **Replies:** 1\ **Last updated:** [September 20, 2022, 12:06pm UTC](https://discuss.elastic.co/t/i-m-unable-to-restart-the-logstash-after-updating-the-conf-d-file/314768 "2022-09-20T12:06:02Z")
Im unable to start logstash after updating the conf.d file in logstash. i checked the logs it is showing as Attempted to resurrect connection to dead ES instance, but got an error. I need to start the logstash now. --- ## [Logstash JDBC - All rows run only once](https://discuss.elastic.co/t/logstash-jdbc-all-rows-run-only-once/314756)
**Author:** [@tag\_v](https://discuss.elastic.co/u/tag_v)\ **Replies:** 0\ **Last updated:** [September 20, 2022, 8:54am UTC](https://discuss.elastic.co/t/logstash-jdbc-all-rows-run-only-once/314756 "2022-09-20T08:54:53Z")
I want to migrate DB table data with millions of rows to S3. I found jdbc plugin. But not clear how to make it run only once for all rows paginated way - The main question is on Scheduling: I don't want to run repeatedly… --- ## [Logstash filter string anywhere](https://discuss.elastic.co/t/logstash-filter-string-anywhere/314552)
**Author:** [@moberreiter](https://discuss.elastic.co/u/moberreiter)\ **Replies:** 5\ **Last updated:** [September 20, 2022, 8:37am UTC](https://discuss.elastic.co/t/logstash-filter-string-anywhere/314552 "2022-09-20T08:37:32Z")
Hi there, I want to filter firewall logs if some specific string matches anywhere in the message and do not know exactly how to do it. Sample Log message: {"zone\_src":"SOURCE","zone\_dst":"EXTERNAL","reason":"rule","ru… --- ## [Logstash S3 output prefix - Event field timestamp](https://discuss.elastic.co/t/logstash-s3-output-prefix-event-field-timestamp/314708)
**Author:** [@tag\_v](https://discuss.elastic.co/u/tag_v)\ **Replies:** 2\ **Last updated:** [September 20, 2022, 8:21am UTC](https://discuss.elastic.co/t/logstash-s3-output-prefix-event-field-timestamp/314708 "2022-09-20T08:21:48Z")
How to set Logstash S3 output prefix dynamically with an event field value in format: "%{+YYYY}/%{+MM}/%{+dd}/%{+HH}" ? input: {"record\_time":"2017-03-09T04:07:51.520Z"} required s3 prefix: 2017/03/09/04 --- ## [Need to Parse a nested JSON message in #Logstash](https://discuss.elastic.co/t/need-to-parse-a-nested-json-message-in-logstash/312979)
**Author:** [@pavanKumar2K](https://discuss.elastic.co/u/pavanKumar2K)\ **Replies:** 2\ **Last updated:** [September 20, 2022, 7:29am UTC](https://discuss.elastic.co/t/need-to-parse-a-nested-json-message-in-logstash/312979 "2022-09-20T07:29:00Z")
Hello , I am trying to send my logs files ( .txt / json files ) to Logstash via Filebeat my sample log structure is as below : {"LogDetails":{"transaction-id":"1234","channel-id":"abc","APIName":"testapi","OperationNa… --- ## [How do I parse CEF messages comprises of json fields in between?](https://discuss.elastic.co/t/how-do-i-parse-cef-messages-comprises-of-json-fields-in-between/314718)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 4\ **Last updated:** [September 20, 2022, 6:36am UTC](https://discuss.elastic.co/t/how-do-i-parse-cef-messages-comprises-of-json-fields-in-between/314718 "2022-09-20T06:36:03Z")
Hi Team, I am accepting Cisco Ironport messages into elasticsearch using logstash cef plugin. However certain messages comprises of json fields in between and those are not getting parsed. Can someone please help me abo… --- ## ["long objects" json pattern regular expression logstash](https://discuss.elastic.co/t/long-objects-json-pattern-regular-expression-logstash/314700)
**Author:** [@Vitoria\_De\_Lara](https://discuss.elastic.co/u/Vitoria_De_Lara)\ **Replies:** 0\ **Last updated:** [September 19, 2022, 1:56pm UTC](https://discuss.elastic.co/t/long-objects-json-pattern-regular-expression-logstash/314700 "2022-09-19T13:56:02Z")
I have a long json file and I need to capture it in parts for elasticsearch --- ## [Connection Refused](https://discuss.elastic.co/t/connection-refused/313418)
**Author:** [@ombit](https://discuss.elastic.co/u/ombit)\ **Replies:** 10\ **Last updated:** [September 19, 2022, 1:08pm UTC](https://discuss.elastic.co/t/connection-refused/313418 "2022-09-19T13:08:23Z")
Hi all, I need to create a solution that amends syslogs to be RFC3164 compliant and ultimately send them on to a QRADAR at a customer's site. To test this I have created two linux VMs both of which can ping each other… --- ## [Logstash number of threads keeps increasing until crash](https://discuss.elastic.co/t/logstash-number-of-threads-keeps-increasing-until-crash/313463)
**Author:** [@Alain\_Bod](https://discuss.elastic.co/u/Alain_Bod)\ **Replies:** 13\ **Last updated:** [September 19, 2022, 1:06pm UTC](https://discuss.elastic.co/t/logstash-number-of-threads-keeps-increasing-until-crash/313463 "2022-09-19T13:06:06Z")
Hi, I'm testing the following pipeline (stack version 8.4.1): input { google\_cloud\_storage {...} } filter {csv {...} mutate {...} } output { elasticsearch {...} } with a large number of events (3M+). After a few min… --- ## [KV filter on ugly json log](https://discuss.elastic.co/t/kv-filter-on-ugly-json-log/314575)
**Author:** [@stanley783](https://discuss.elastic.co/u/stanley783)\ **Replies:** 4\ **Last updated:** [September 19, 2022, 10:43am UTC](https://discuss.elastic.co/t/kv-filter-on-ugly-json-log/314575 "2022-09-19T10:43:07Z")
Hi, received log in json format, json filter parsed correctly but one field, "details", is in a unfinished/incorrect format lets say and json cant handle it, therefore i applied kv but cant get desired outcome - correct… --- ## [\[LogStash::Runner\] runner - Logstash shut down](https://discuss.elastic.co/t/logstash-runner-runner-logstash-shut-down/314612)
**Author:** [@Veera\_V](https://discuss.elastic.co/u/Veera_V)\ **Replies:** 2\ **Last updated:** [September 17, 2022, 1:16pm UTC](https://discuss.elastic.co/t/logstash-runner-runner-logstash-shut-down/314612 "2022-09-17T13:16:41Z")
Hi there, I'm trying to connect logstash and Opensearch, but I'm getting error "Logstash::Runner\] runner - Logstash shutdown. Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. … --- ## [Break down log messages to keywords](https://discuss.elastic.co/t/break-down-log-messages-to-keywords/314588)
**Author:** [@Mark\_S](https://discuss.elastic.co/u/Mark_S)\ **Replies:** 0\ **Last updated:** [September 16, 2022, 3:39pm UTC](https://discuss.elastic.co/t/break-down-log-messages-to-keywords/314588 "2022-09-16T15:39:40Z")
I want to index logs from different applications in one index. However some applications don't use exactly the same log format.... How can I use a grok pattern in the input section of the config file to break down the … --- ## [Logstash from SQL Server to Elasticsearch character encoding problem](https://discuss.elastic.co/t/logstash-from-sql-server-to-elasticsearch-character-encoding-problem/314587)
**Author:** [@Startech](https://discuss.elastic.co/u/Startech)\ **Replies:** 0\ **Last updated:** [September 16, 2022, 3:23pm UTC](https://discuss.elastic.co/t/logstash-from-sql-server-to-elasticsearch-character-encoding-problem/314587 "2022-09-16T15:23:25Z")
Hi, I am using ELK stack v8.4.1 and trying to integrate data between SQL Server and Elasticsearch via Logstash. My source table includes Turkish characters (collation SQL\_Latin1\_General\_CP1\_CI\_AS). When Logstash writes … --- ## [How to run a pipeline from a pipeline](https://discuss.elastic.co/t/how-to-run-a-pipeline-from-a-pipeline/314562)
**Author:** [@reed](https://discuss.elastic.co/u/reed)\ **Replies:** 4\ **Last updated:** [September 16, 2022, 2:37pm UTC](https://discuss.elastic.co/t/how-to-run-a-pipeline-from-a-pipeline/314562 "2022-09-16T14:37:30Z")
Hi All, in my project I have 3 pipelines, I know that a pipeline can be scheduled but in my case I need to run the third one at the end of other two. There is a way to do this? run the pipelines in order: pipeline1 -\>… --- ## [UDP Listener died, address already in use](https://discuss.elastic.co/t/udp-listener-died-address-already-in-use/314281)
**Author:** [@sirineb](https://discuss.elastic.co/u/sirineb)\ **Replies:** 4\ **Last updated:** [September 16, 2022, 2:24pm UTC](https://discuss.elastic.co/t/udp-listener-died-address-already-in-use/314281 "2022-09-16T14:24:49Z")
Hi, I have two pipeline configurations located in /etc/logstash/conf.d. I can make work both seperately by running /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/\*\*.conf -r, and it sends data to elasticsearch … --- ## [Issue with logstash agent](https://discuss.elastic.co/t/issue-with-logstash-agent/314578)
**Author:** [@yas](https://discuss.elastic.co/u/yas)\ **Replies:** 0\ **Last updated:** [September 16, 2022, 1:34pm UTC](https://discuss.elastic.co/t/issue-with-logstash-agent/314578 "2022-09-16T13:34:02Z")
Having an issue with Logstash agents ( linux hosts ) sending files to a server - what seems to happen is on the agent starting up , the first few rows of the logs are sent after which it just stops and nothing further is… --- ## [Process events from split](https://discuss.elastic.co/t/process-events-from-split/314564)
**Author:** [@sirReeall](https://discuss.elastic.co/u/sirReeall)\ **Replies:** 1\ **Last updated:** [September 16, 2022, 10:01am UTC](https://discuss.elastic.co/t/process-events-from-split/314564 "2022-09-16T10:01:34Z")
Is it possible to have logstash process the events generated from a split? For example, give the following: 2022-09-06 23:39:01.034+0000 INFO \[my.java.class\] Process started \\n some lines \\n some more lines \\n … --- ## [Create elastic field alias](https://discuss.elastic.co/t/create-elastic-field-alias/313966)
**Author:** [@sirReeall](https://discuss.elastic.co/u/sirReeall)\ **Replies:** 4\ **Last updated:** [September 16, 2022, 9:25am UTC](https://discuss.elastic.co/t/create-elastic-field-alias/313966 "2022-09-16T09:25:34Z")
Hello, Is it possible for logstash to alias fields as described in the elastic docs below? --- ## [Invalid Index Name-Index name must be lowercase error](https://discuss.elastic.co/t/invalid-index-name-index-name-must-be-lowercase-error/314287)
**Author:** [@tapas](https://discuss.elastic.co/u/tapas)\ **Replies:** 15\ **Last updated:** [September 16, 2022, 7:55am UTC](https://discuss.elastic.co/t/invalid-index-name-index-name-must-be-lowercase-error/314287 "2022-09-16T07:55:52Z")
I am getting error in logstash logs saying that the index is invalid and reason is it should be lower case.but the index name in the log is different from the index i declared in the logstash.config file.in log file it i… --- ## [Logstash filter taking. high cpu](https://discuss.elastic.co/t/logstash-filter-taking-high-cpu/314550)
**Author:** [@krishan.kumar](https://discuss.elastic.co/u/krishan.kumar)\ **Replies:** 0\ **Last updated:** [September 16, 2022, 6:35am UTC](https://discuss.elastic.co/t/logstash-filter-taking-high-cpu/314550 "2022-09-16T06:35:54Z")
Hi, My logstash grok pattern taking high CPU This is my log format I, \[2022-09-16T06:27:40.386003 #13375\] INFO -- : \[s45g0-e726-56y7-adcf-3382276c1a77\] \[user\_id: xxxxxx, portfolio\_id: xxxxxxx\] {"method":"GET","path":… --- ## [Logstash cipher fragment error](https://discuss.elastic.co/t/logstash-cipher-fragment-error/314523)
**Author:** [@RaonyO](https://discuss.elastic.co/u/RaonyO)\ **Replies:** 3\ **Last updated:** [September 15, 2022, 3:41pm UTC](https://discuss.elastic.co/t/logstash-cipher-fragment-error/314523 "2022-09-15T15:41:03Z")
I have an error in a pipeline in logstash, the input is via tcp and ssl. config: input { tcp { port =\> 7771 codec =\> line ssl\_verify =\> false ssl\_enable =\> true ssl\_cert =\> "/etc/logstash/certs/cert.crt… --- ## [How is the state (last read file or position in a file) is maintained for multiple pods running logstash](https://discuss.elastic.co/t/how-is-the-state-last-read-file-or-position-in-a-file-is-maintained-for-multiple-pods-running-logstash/314502)
**Author:** [@Jyoti\_Patel](https://discuss.elastic.co/u/Jyoti_Patel)\ **Replies:** 4\ **Last updated:** [September 15, 2022, 2:12pm UTC](https://discuss.elastic.co/t/how-is-the-state-last-read-file-or-position-in-a-file-is-maintained-for-multiple-pods-running-logstash/314502 "2022-09-15T14:12:00Z")
Hi Everyone, I just have a query regarding last read s3 file or the last read line in the s3 file. How or where is that data stored? and how is that shared among multiple pods running logstash? What if the pod is repl… --- ## [Logstash and filebeat data streams](https://discuss.elastic.co/t/logstash-and-filebeat-data-streams/314488)
**Author:** [@gborg](https://discuss.elastic.co/u/gborg)\ **Replies:** 1\ **Last updated:** [September 15, 2022, 1:24pm UTC](https://discuss.elastic.co/t/logstash-and-filebeat-data-streams/314488 "2022-09-15T13:24:35Z")
Hello I have a setup where I run filebeats, which send their logs to a queue, logstash reads off the queue and writes the data to elasticsearch. With version 7 I had no issues, I recently upgraded to elasticsearch to e… --- ## [Logic Error after ingesting number](https://discuss.elastic.co/t/logic-error-after-ingesting-number/314331)
**Author:** [@hnf](https://discuss.elastic.co/u/hnf)\ **Replies:** 7\ **Last updated:** [September 15, 2022, 8:32am UTC](https://discuss.elastic.co/t/logic-error-after-ingesting-number/314331 "2022-09-15T08:32:38Z")
I have an xlsx file that I convert in CSV file using ssconvert tool. In the new generated file, there's one field in floating point type. But after ingestion, the values taken from the file is no more in float. For exem… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=113) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=115)