# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=98 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 99 --- ## [Logstash pipeline worker stops processing events because of exception](https://discuss.elastic.co/t/logstash-pipeline-worker-stops-processing-events-because-of-exception/322126)
**Author:** [@tcapp24](https://discuss.elastic.co/u/tcapp24)\ **Replies:** 3\ **Last updated:** [December 29, 2022, 1:19am UTC](https://discuss.elastic.co/t/logstash-pipeline-worker-stops-processing-events-because-of-exception/322126 "2022-12-29T01:19:59Z")
Hello, Currently our beats-dead-letter-queue-processing-pipeline is causing a exception which makes events stop outputting to Elasticsearch: beats-dead-letter-queue-processing-pipeline\] Pipeline worker error, the pipe… --- ## [Logstash Filter for Cisco ASA Source Destination and Communication](https://discuss.elastic.co/t/logstash-filter-for-cisco-asa-source-destination-and-communication/322063)
**Author:** [@Pratik\_Srivastava](https://discuss.elastic.co/u/Pratik_Srivastava)\ **Replies:** 5\ **Last updated:** [December 28, 2022, 9:41pm UTC](https://discuss.elastic.co/t/logstash-filter-for-cisco-asa-source-destination-and-communication/322063 "2022-12-28T21:41:31Z")
I want to configure logstash filter to see asa logs with communication detail. LOG Sample: Dec 27 02:48:08 Test-FW : %ASA-6-302014: Teardown TCP connection 3505833084 for Test-OUT:192.168.1.10/58538 to Mgmt-IN:192.168.1… --- ## [How to troubleshoot "undefined method \`+' for nil:NilClass"](https://discuss.elastic.co/t/how-to-troubleshoot-undefined-method-for-nil-nilclass/322121)
**Author:** [@INS](https://discuss.elastic.co/u/INS)\ **Replies:** 3\ **Last updated:** [December 28, 2022, 8:54pm UTC](https://discuss.elastic.co/t/how-to-troubleshoot-undefined-method-for-nil-nilclass/322121 "2022-12-28T20:54:47Z")
Hi Maybe You have some idea how to troubleshoot in the long pipeline the bug referring to \[2022-12-28T19:47:41,016\]\[ERROR\]\[logstash.filters.ruby \]\[main\]\[1b91d552e5ff1ebff261eab8e51449b8aba605255321bb2b70376384a24931… --- ## [Logstash - Schedule files](https://discuss.elastic.co/t/logstash-schedule-files/322057)
**Author:** [@SilasMuniz1](https://discuss.elastic.co/u/SilasMuniz1)\ **Replies:** 6\ **Last updated:** [December 28, 2022, 1:59am UTC](https://discuss.elastic.co/t/logstash-schedule-files/322057 "2022-12-28T01:59:29Z")
Hello, I need read new files inside one directory. I have a script that every 10 minutes send news files for this directory. I created this setup in my pipeline: When the first files were insert in the directory ev… --- ## [How to create new logstash grok field](https://discuss.elastic.co/t/how-to-create-new-logstash-grok-field/320722)
**Author:** [@anon74213320](https://discuss.elastic.co/u/anon74213320)\ **Replies:** 1\ **Last updated:** [December 27, 2022, 6:35pm UTC](https://discuss.elastic.co/t/how-to-create-new-logstash-grok-field/320722 "2022-12-27T18:35:57Z")
I have an ELK stack Filebeat---\>Logstash----\>Elasticsearch\<----Kibana. I m shipping the logs in a proper way, but I would need to add a custom fields available in Kibana, for searching puproses. The Log file contains a… --- ## [Out of memory error and duplicate rows](https://discuss.elastic.co/t/out-of-memory-error-and-duplicate-rows/321932)
**Author:** [@darius12](https://discuss.elastic.co/u/darius12)\ **Replies:** 4\ **Last updated:** [December 27, 2022, 2:29pm UTC](https://discuss.elastic.co/t/out-of-memory-error-and-duplicate-rows/321932 "2022-12-27T14:29:06Z")
getting this error when trying to index using logstash: warning: thread "\[main\]\>worker1" terminated with exception (report\_on\_exception is true): java.lang.OutOfMemoryError: UTF16 String size is 1371255266, should be l… --- ## [Using jdbc in logstash to import Elasticsearch data Connection error](https://discuss.elastic.co/t/using-jdbc-in-logstash-to-import-elasticsearch-data-connection-error/322037)
**Author:** [@Sandeep\_Raju](https://discuss.elastic.co/u/Sandeep_Raju)\ **Replies:** 1\ **Last updated:** [December 27, 2022, 12:37pm UTC](https://discuss.elastic.co/t/using-jdbc-in-logstash-to-import-elasticsearch-data-connection-error/322037 "2022-12-27T12:37:27Z")
Hi all, I'm trying to import data from different databases from different clusters to my elasticsearch database. Before importing mysql data to my elasticsearch database , I'm trying to import elasticsearch database fr… --- ## [Logstash - Aggregate xpath data](https://discuss.elastic.co/t/logstash-aggregate-xpath-data/321567)
**Author:** [@Basti97](https://discuss.elastic.co/u/Basti97)\ **Replies:** 8\ **Last updated:** [December 21, 2022, 1:14pm UTC](https://discuss.elastic.co/t/logstash-aggregate-xpath-data/321567 "2022-12-21T13:14:29Z")
Hello, I am a Logstash beginner and have probably a relatively simple question. Im using xpath to get data from a xml document (have shortened the code on purpose): filter { xml { source =\> "message" store\_xml =\> f… --- ## [Adding items to a new field](https://discuss.elastic.co/t/adding-items-to-a-new-field/321995)
**Author:** [@Glad](https://discuss.elastic.co/u/Glad)\ **Replies:** 0\ **Last updated:** [December 26, 2022, 4:15pm UTC](https://discuss.elastic.co/t/adding-items-to-a-new-field/321995 "2022-12-26T16:15:31Z")
Hi, I recently started using Elastic and I'm having a problem. I am using a monitoring system for my tests to retrieve various data from the machines I am using. This monitoring system has an api that allows Logstash to … --- ## [Extract timestamp from multiple records](https://discuss.elastic.co/t/extract-timestamp-from-multiple-records/321993)
**Author:** [@mail2shanth](https://discuss.elastic.co/u/mail2shanth)\ **Replies:** 0\ **Last updated:** [December 26, 2022, 3:38pm UTC](https://discuss.elastic.co/t/extract-timestamp-from-multiple-records/321993 "2022-12-26T15:38:45Z")
Hi, Below is my config file, input { file{ path =\> "/Users/.../Work/Projects/ELK/Logstash/Input/\*.txt" start\_position =\> beginning codec =\> "json" type =\> "data" sincedb\_path =\> "NUL" } } filter { … --- ## [Index.lifecycle.rollover\_alias \[filebeat-7.17.7\] does not point to index \[filebeat-7.17.7-2022.12.21\]](https://discuss.elastic.co/t/index-lifecycle-rollover-alias-filebeat-7-17-7-does-not-point-to-index-filebeat-7-17-7-2022-12-21/321917)
**Author:** [@tjay](https://discuss.elastic.co/u/tjay)\ **Replies:** 14\ **Last updated:** [December 26, 2022, 11:40am UTC](https://discuss.elastic.co/t/index-lifecycle-rollover-alias-filebeat-7-17-7-does-not-point-to-index-filebeat-7-17-7-2022-12-21/321917 "2022-12-26T11:40:19Z")
Hi, I started using elasticsearch, kibana, logstash and filebeat and noticed that the rollover error is occurring. Can someone shed a light on this? It looks there are two indexes and the second one (000002) is used at… --- ## [Logstash filter not working](https://discuss.elastic.co/t/logstash-filter-not-working/321948)
**Author:** [@parisila](https://discuss.elastic.co/u/parisila)\ **Replies:** 2\ **Last updated:** [December 26, 2022, 8:29am UTC](https://discuss.elastic.co/t/logstash-filter-not-working/321948 "2022-12-26T08:29:36Z")
I have filebeat, logstash and elasticsearch. My logstash config.d looks like this 02-beats-input.conf receives the message from filebeat 30-elasticsearch-output.conf sends to elasticsearch ingest pipeline. Logstash… --- ## [Trouble with filter conditional logic](https://discuss.elastic.co/t/trouble-with-filter-conditional-logic/321930)
**Author:** [@willdennis](https://discuss.elastic.co/u/willdennis)\ **Replies:** 5\ **Last updated:** [December 24, 2022, 3:53am UTC](https://discuss.elastic.co/t/trouble-with-filter-conditional-logic/321930 "2022-12-24T03:53:21Z")
I have had the following filter conditional logic in place for a while now, and it's working well: filter { mutate { \[...\] } if "STRING1" in \[message\] { \[...\] } else if \[type\] == "syslog" { \[...\] } … --- ## [How do I use prune inside of a specific field?](https://discuss.elastic.co/t/how-do-i-use-prune-inside-of-a-specific-field/321950)
**Author:** [@ste1](https://discuss.elastic.co/u/ste1)\ **Replies:** 1\ **Last updated:** [December 24, 2022, 3:10am UTC](https://discuss.elastic.co/t/how-do-i-use-prune-inside-of-a-specific-field/321950 "2022-12-24T03:10:54Z")
I have a bunch of csv files from which I need to extract the "user" and "hwid" columns from. The csv filter will pull out all of the columns and put them in a field called "data". I then use the prune filter to keep only… --- ## [Problem "grok"ing when there is a conditional part on it](https://discuss.elastic.co/t/problem-grok-ing-when-there-is-a-conditional-part-on-it/321919)
**Author:** [@syunusic](https://discuss.elastic.co/u/syunusic)\ **Replies:** 4\ **Last updated:** [December 23, 2022, 11:53pm UTC](https://discuss.elastic.co/t/problem-grok-ing-when-there-is-a-conditional-part-on-it/321919 "2022-12-23T23:53:54Z")
If I have this text: blah1=faa faa2 blah2=fee blah3=fii blah4=foo how can I have the values of each variable (the one that are before the equal sign) with regex? I’ve tried with this grok: blah1=(?\\[^=\]+) blah2=… --- ## [How to dinamically replace - with \_ in nested field names](https://discuss.elastic.co/t/how-to-dinamically-replace-with-in-nested-field-names/321936)
**Author:** [@anubisg1](https://discuss.elastic.co/u/anubisg1)\ **Replies:** 10\ **Last updated:** [December 23, 2022, 9:02pm UTC](https://discuss.elastic.co/t/how-to-dinamically-replace-with-in-nested-field-names/321936 "2022-12-23T21:02:54Z")
i'm try to perform a field name replacement dynamically. I do not know ahead of time how many fields nr what they are. i found this in an elasticsearch pipeline but i need to do the exact same in logstash # Converts a… --- ## [If condition list in field](https://discuss.elastic.co/t/if-condition-list-in-field/321886)
**Author:** [@mikhatanu](https://discuss.elastic.co/u/mikhatanu)\ **Replies:** 1\ **Last updated:** [December 23, 2022, 6:02pm UTC](https://discuss.elastic.co/t/if-condition-list-in-field/321886 "2022-12-23T18:02:43Z")
hello, is it possible to do a searching of a list in field? e.g. filter{ if \["a","b","c","d"\] in \[message\]{ mutate{ add\_field=\>{"new\_field"=\>"%{message}"} } } } i want the "new\_field… --- ## [Using the geoip filter with commercial databases](https://discuss.elastic.co/t/using-the-geoip-filter-with-commercial-databases/321874)
**Author:** [@patam](https://discuss.elastic.co/u/patam)\ **Replies:** 2\ **Last updated:** [December 23, 2022, 3:11pm UTC](https://discuss.elastic.co/t/using-the-geoip-filter-with-commercial-databases/321874 "2022-12-23T15:11:52Z")
I am trying to use the commercial Anonymous IP database for the geoip filter and I'm not sure how to retreive the information from the database. I have Logstash updating the db and the right .mmdb is there I am just unsu… --- ## [What network mode use in podman-compose](https://discuss.elastic.co/t/what-network-mode-use-in-podman-compose/321914)
**Author:** [@vnovotny98](https://discuss.elastic.co/u/vnovotny98)\ **Replies:** 5\ **Last updated:** [December 23, 2022, 1:19pm UTC](https://discuss.elastic.co/t/what-network-mode-use-in-podman-compose/321914 "2022-12-23T13:19:02Z")
Hello, I have a problem with podman-compose. I have running ELK stack and communication between apps is working but I cant figure out why I cant connect from another host to logstash input. Can you please tell me which… --- ## [How can we communicate between different container of elk stacks if published in kubernetes?](https://discuss.elastic.co/t/how-can-we-communicate-between-different-container-of-elk-stacks-if-published-in-kubernetes/321908)
**Author:** [@Rohit\_Kundu](https://discuss.elastic.co/u/Rohit_Kundu)\ **Replies:** 0\ **Last updated:** [December 23, 2022, 11:06am UTC](https://discuss.elastic.co/t/how-can-we-communicate-between-different-container-of-elk-stacks-if-published-in-kubernetes/321908 "2022-12-23T11:06:08Z")
I want to create docker images of each applications of elk stacks, and will later publish on kubernetes. Wanted to know that how can we communicate between different containers of elk --- ## [Logstash JDBC input plugin: Java::OrgPostgresqlUtil::PSQLException: An I/O error occurred while sending to the backend](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-java-an-i-o-error-occurred-while-sending-to-the-backend/321900)
**Author:** [@Thijsvdp](https://discuss.elastic.co/u/Thijsvdp)\ **Replies:** 0\ **Last updated:** [December 23, 2022, 10:12am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-java-an-i-o-error-occurred-while-sending-to-the-backend/321900 "2022-12-23T10:12:05Z")
Hi all, I have created a Logstash pipeline which aims to sync an Elasticsearch index with a Postgres database. Unfortunately, there is a query that does not perform well and takes quite some time. Once every now and the… --- ## [Parsing CSV with different header](https://discuss.elastic.co/t/parsing-csv-with-different-header/321887)
**Author:** [@SKiD](https://discuss.elastic.co/u/SKiD)\ **Replies:** 0\ **Last updated:** [December 23, 2022, 8:07am UTC](https://discuss.elastic.co/t/parsing-csv-with-different-header/321887 "2022-12-23T08:07:07Z")
Hello, I have some thousand CSV files which I need to index into elasticsearch. All of those CSV files have different headers. I tried to use the CSV filter for logstash, but then I read THAT. Besides of my incomprehens… --- ## [Multiple logstash instances consume kafka, only one is working](https://discuss.elastic.co/t/multiple-logstash-instances-consume-kafka-only-one-is-working/321880)
**Author:** [@TemplateXu](https://discuss.elastic.co/u/TemplateXu)\ **Replies:** 1\ **Last updated:** [December 23, 2022, 6:45am UTC](https://discuss.elastic.co/t/multiple-logstash-instances-consume-kafka-only-one-is-working/321880 "2022-12-23T06:45:47Z")
Multiple logstash instances consume log messages in kafka, but only one of them is working. When I close one of them, the other starts to work. How to make two logstash instances work together --- ## [Parse different time duration formats/units to common format](https://discuss.elastic.co/t/parse-different-time-duration-formats-units-to-common-format/321742)
**Author:** [@wespe](https://discuss.elastic.co/u/wespe)\ **Replies:** 4\ **Last updated:** [December 22, 2022, 7:49am UTC](https://discuss.elastic.co/t/parse-different-time-duration-formats-units-to-common-format/321742 "2022-12-22T07:49:27Z")
Hi there, Part of my input json looks as follows: "phaseTimes": { "authorize": "28.201µs", "filter": "27.522068ms", "indexScan": "2.004642056s", "instantiate": "40.002µs", "run": "2.041368619s" … --- ## [What's the difference between s3 input in filebeat and s3 input in logstash](https://discuss.elastic.co/t/whats-the-difference-between-s3-input-in-filebeat-and-s3-input-in-logstash/321798)
**Author:** [@stwang](https://discuss.elastic.co/u/stwang)\ **Replies:** 1\ **Last updated:** [December 22, 2022, 12:05am UTC](https://discuss.elastic.co/t/whats-the-difference-between-s3-input-in-filebeat-and-s3-input-in-logstash/321798 "2022-12-22T00:05:43Z")
Hi There, We try to ingest data from s3 to Elasticsearch, and I know filebeat and logstash all support S3 input. Which one should I use, is there has any guidence, in which situation to use filebeat or logstash. Cheer… --- ## [Update existing record in elasticsearch while ingesting data through logstash pipeline](https://discuss.elastic.co/t/update-existing-record-in-elasticsearch-while-ingesting-data-through-logstash-pipeline/321769)
**Author:** [@Tanvi07](https://discuss.elastic.co/u/Tanvi07)\ **Replies:** 0\ **Last updated:** [December 21, 2022, 1:10pm UTC](https://discuss.elastic.co/t/update-existing-record-in-elasticsearch-while-ingesting-data-through-logstash-pipeline/321769 "2022-12-21T13:10:24Z")
Hi, I have a requirement where I need to ingest data from mysql db into elasticsearch, for this i have made use of logstash jdbc input plugin. My data is in large volume and i need to implement incremental ingestion now… --- ## [Logstash problem with disabling ecs\_compatibility](https://discuss.elastic.co/t/logstash-problem-with-disabling-ecs-compatibility/321740)
**Author:** [@Anonym123](https://discuss.elastic.co/u/Anonym123)\ **Replies:** 2\ **Last updated:** [December 21, 2022, 12:18pm UTC](https://discuss.elastic.co/t/logstash-problem-with-disabling-ecs-compatibility/321740 "2022-12-21T12:18:06Z")
logstash image is: 7.17.3 with the prune plugin This is my logstash.conf pipeline.ecs\_compatibility: disabled input { beats { port =\> "XXXXXX" ecs\_compatibility =\> disabled } } … --- ## [Logstash-output-syslog never writes anything to local syslog on REL8 in podman with podman-compose](https://discuss.elastic.co/t/logstash-output-syslog-never-writes-anything-to-local-syslog-on-rel8-in-podman-with-podman-compose/321697)
**Author:** [@Dale\_Bingham](https://discuss.elastic.co/u/Dale_Bingham)\ **Replies:** 5\ **Last updated:** [December 21, 2022, 12:07pm UTC](https://discuss.elastic.co/t/logstash-output-syslog-never-writes-anything-to-local-syslog-on-rel8-in-podman-with-podman-compose/321697 "2022-12-21T12:07:40Z")
I have ELK in my podman-compose setup and I can push any logs I get to Logstash through to Elasticsearch just fine. It is when I am trying to also send to syslog for those that already have a process to pull all syslog d… --- ## [Logstash Scheduled Run of conf file and error alert](https://discuss.elastic.co/t/logstash-scheduled-run-of-conf-file-and-error-alert/321743)
**Author:** [@Sandeep\_Raju](https://discuss.elastic.co/u/Sandeep_Raju)\ **Replies:** 1\ **Last updated:** [December 21, 2022, 11:41am UTC](https://discuss.elastic.co/t/logstash-scheduled-run-of-conf-file-and-error-alert/321743 "2022-12-21T11:41:32Z")
Hi , I'm using logstash to push my CSV data from a location to elasticsearch database. In my project I want a scheduled CSV data push at a particular time. Also want to know if any error is generated while pushing dat… --- ## [Logstash not able to connect to elastic search hosted on asw](https://discuss.elastic.co/t/logstash-not-able-to-connect-to-elastic-search-hosted-on-asw/321737)
**Author:** [@shivam585](https://discuss.elastic.co/u/shivam585)\ **Replies:** 8\ **Last updated:** [December 21, 2022, 9:17am UTC](https://discuss.elastic.co/t/logstash-not-able-to-connect-to-elastic-search-hosted-on-asw/321737 "2022-12-21T09:17:30Z")
\[2022-12-21T07:09:39,189\]\[INFO \]\[logstash.runner \] Log4j configuration path used is: /etc/logstash/log4j2.properties \[2022-12-21T07:09:39,205\]\[INFO \]\[logstash.runner \] Starting Logstash {"logstash.versi… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=97) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=99)