# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=13 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 14 --- ## [Can't make MongoDB connection work in Logstash Filter](https://discuss.elastic.co/t/cant-make-mongodb-connection-work-in-logstash-filter/373144)
**Author:** [@mdschoe](https://discuss.elastic.co/u/mdschoe)\ **Replies:** 2\ **Last updated:** [January 14, 2025, 5:26am UTC](https://discuss.elastic.co/t/cant-make-mongodb-connection-work-in-logstash-filter/373144 "2025-01-14T05:26:14Z")
I've been fighting this issue for days and keep circling back to the same problem: \[2025-01-13T21:24:32,037\]\[ERROR\]\[logstash.javapipeline \]\[main\] Pipeline error {:pipeline\_id=\>"main", :exception=\>#\*\*\ **Author:** [@zerratriani](https://discuss.elastic.co/u/zerratriani)\ **Replies:** 3\ **Last updated:** [January 13, 2025, 8:15am UTC](https://discuss.elastic.co/t/how-to-parsing-nested-json/372879 "2025-01-13T08:15:48Z") So i have log that alreary parse in json. But i want to parsing again that nested? This the log i manipulated cause is sensitive data. And i want to parse again is it inside responsePayload.data {"cif":"XXX","gram":"X… --- ## [Parse single-line json with logstash](https://discuss.elastic.co/t/parse-single-line-json-with-logstash/373096)
**Author:** [@Resort](https://discuss.elastic.co/u/Resort)\ **Replies:** 3\ **Last updated:** [January 12, 2025, 2:48pm UTC](https://discuss.elastic.co/t/parse-single-line-json-with-logstash/373096 "2025-01-12T14:48:21Z")
Hello, I'm looking for assistance with my attempt of passing logs of .json type to Elasticsearch using Logstash. The tricky moment is that the .json file contains one single valid data and is being ignored by Logstash. … --- ## [Parse a log file from esp32](https://discuss.elastic.co/t/parse-a-log-file-from-esp32/373020)
**Author:** [@s0mbra](https://discuss.elastic.co/u/s0mbra)\ **Replies:** 2\ **Last updated:** [January 10, 2025, 5:41pm UTC](https://discuss.elastic.co/t/parse-a-log-file-from-esp32/373020 "2025-01-10T17:41:57Z")
Hi everone! Im starting to develop a project with the ESP32. This project generates a web interface to control outputs on the ESP32. I want to collect the logs and process them through the ELK stack, but I am not being … --- ## [Logstash Health Report API - Not Found](https://discuss.elastic.co/t/logstash-health-report-api-not-found/372952)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 5\ **Last updated:** [January 10, 2025, 4:40pm UTC](https://discuss.elastic.co/t/logstash-health-report-api-not-found/372952 "2025-01-10T16:40:24Z")
ELK version : 8.15 Below API is giving the Not Found Output . curl -XGET 'localhost:9600/\_health\_report?pretty' While below api is working and giving output curl -XGET 'localhost:9600/?pretty' --- ## [Update Logstash Keystore while it's running as a Service](https://discuss.elastic.co/t/update-logstash-keystore-while-its-running-as-a-service/323268)
**Author:** [@Hichem](https://discuss.elastic.co/u/Hichem)\ **Replies:** 2\ **Last updated:** [January 10, 2025, 4:33pm UTC](https://discuss.elastic.co/t/update-logstash-keystore-while-its-running-as-a-service/323268 "2025-01-10T16:33:24Z")
Is there is any recommendation on how to make changes to Logstash keystore variables (update variable or add new ones) while the service is running. Do we need to stop the service before making changes? Does it require… --- ## [Java not found when executing script that needs Java](https://discuss.elastic.co/t/java-not-found-when-executing-script-that-needs-java/373058)
**Author:** [@nilsen](https://discuss.elastic.co/u/nilsen)\ **Replies:** 3\ **Last updated:** [January 10, 2025, 3:17pm UTC](https://discuss.elastic.co/t/java-not-found-when-executing-script-that-needs-java/373058 "2025-01-10T15:17:46Z")
After upgrading to Logstash 8.11.3, our Logstash stopped working because it can no longer execute a script that needs Java. I believe this is caused by Logstash now using it's internally shipped JDK. Logstash Exec Input… --- ## [How to set Logstash worker](https://discuss.elastic.co/t/how-to-set-logstash-worker/373021)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 3\ **Last updated:** [January 9, 2025, 7:19pm UTC](https://discuss.elastic.co/t/how-to-set-logstash-worker/373021 "2025-01-09T19:19:37Z")
ELK version : 8.15 We have 2 Logstash instances having 32gb of heap size per node and 6 CPU per node. We have 5 pipeline IDs in both these nodes . For one pipeline id we have set 6 worker , second pipeline id 3 and for… --- ## [S3 input plugin problem with %](https://discuss.elastic.co/t/s3-input-plugin-problem-with/372928)
**Author:** [@Martin\_IT](https://discuss.elastic.co/u/Martin_IT)\ **Replies:** 4\ **Last updated:** [January 9, 2025, 4:22pm UTC](https://discuss.elastic.co/t/s3-input-plugin-problem-with/372928 "2025-01-09T16:22:58Z")
Hi, we use S3 input plugin and we have a problem with backup of indexed files. Our files contain character % because URL encoded format. Indexing is OK but than when file should be moved to the archive we receive errors: … --- ## [Splitting message using Logstash mutate filter](https://discuss.elastic.co/t/splitting-message-using-logstash-mutate-filter/373011)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 0\ **Last updated:** [January 9, 2025, 3:49pm UTC](https://discuss.elastic.co/t/splitting-message-using-logstash-mutate-filter/373011 "2025-01-09T15:49:58Z")
Hi All, The incoming feed (log) to Logstash has parameters which are delimited by ~|~ These are being mutated and split in Logstash as follows. : if \[type\] == "tv\_dmz\_access" { mutate { … --- ## [Logs Not Appearing in Kibana After Instance Refresh Activity](https://discuss.elastic.co/t/logs-not-appearing-in-kibana-after-instance-refresh-activity/372956)
**Author:** [@ErGeek](https://discuss.elastic.co/u/ErGeek)\ **Replies:** 0\ **Last updated:** [January 8, 2025, 8:44pm UTC](https://discuss.elastic.co/t/logs-not-appearing-in-kibana-after-instance-refresh-activity/372956 "2025-01-08T20:44:33Z")
Our log management pipeline operates as follows: Beats -\> MSK (Kafka) -\> Logstash (hosted in EC2) -\> Elasticsearch -\> Kibana. We handle updates to the Logstash servers through the Instance Refresh activity in AWS. This … --- ## [Logstash's Elasticsearch filter plugin bottleneck](https://discuss.elastic.co/t/logstashs-elasticsearch-filter-plugin-bottleneck/372666)
**Author:** [@lethargicsnailspider](https://discuss.elastic.co/u/lethargicsnailspider)\ **Replies:** 11\ **Last updated:** [January 8, 2025, 4:34pm UTC](https://discuss.elastic.co/t/logstashs-elasticsearch-filter-plugin-bottleneck/372666 "2025-01-08T16:34:29Z")
I have a Logstash pipeline set up with multiple Elasticsearch filter plugins to do a lookup of field values ingested via an input filter. Processing and writing the events into Elasticsearch is around 500,000 events per … --- ## [Logstash JDBC\_INPUT does not push data upon it starts when there's scheduler in its config](https://discuss.elastic.co/t/logstash-jdbc-input-does-not-push-data-upon-it-starts-when-theres-scheduler-in-its-config/372870)
**Author:** [@mxu](https://discuss.elastic.co/u/mxu)\ **Replies:** 2\ **Last updated:** [January 7, 2025, 12:03am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-does-not-push-data-upon-it-starts-when-theres-scheduler-in-its-config/372870 "2025-01-07T00:03:18Z")
I have following jdbc\_input configured and expect it runs upon logstash starts AND as scheduled as well. However, it did not run upon logstash starts. I found following old post. It shall run. I have losgtash 8. 13.2. … --- ## [Using join field](https://discuss.elastic.co/t/using-join-field/372753)
**Author:** [@denisbik349](https://discuss.elastic.co/u/denisbik349)\ **Replies:** 0\ **Last updated:** [January 3, 2025, 3:22pm UTC](https://discuss.elastic.co/t/using-join-field/372753 "2025-01-03T15:22:39Z")
Hello everyone! I have problem joining 2 documents in one index by using join type field. I have input, filter and output files configured for Logstash. I have index and index pattern set up in Elasticsearch. I was abl… --- ## [JSON filter plugin](https://discuss.elastic.co/t/json-filter-plugin/372731)
**Author:** [@shaddow](https://discuss.elastic.co/u/shaddow)\ **Replies:** 1\ **Last updated:** [January 3, 2025, 1:08pm UTC](https://discuss.elastic.co/t/json-filter-plugin/372731 "2025-01-03T13:08:45Z")
I want to configure a JSON filter plugin in my ELK stack, but after configuring the following files, I don't see any data in Kibana. I'm not sure if I need to configure any additional files. Can anyone help me? logsta… --- ## [ELK 8, Zabbix 7.2, set trigger on Zabbix server](https://discuss.elastic.co/t/elk-8-zabbix-7-2-set-trigger-on-zabbix-server/372681)
**Author:** [@Mostafa\_Faridi](https://discuss.elastic.co/u/Mostafa_Faridi)\ **Replies:** 1\ **Last updated:** [January 2, 2025, 7:06pm UTC](https://discuss.elastic.co/t/elk-8-zabbix-7-2-set-trigger-on-zabbix-server/372681 "2025-01-02T19:06:13Z")
I have ELK 8 server, on this server I have elasticsearch, logstash and kibana, on other servers I install filebeat, filebeat send logs to logstash, I also have Zabbix 7.2 LTS version too, this is sample logs generate b… --- ## [Grok pattern](https://discuss.elastic.co/t/grok-pattern/372704)
**Author:** [@Eshwar\_K](https://discuss.elastic.co/u/Eshwar_K)\ **Replies:** 2\ **Last updated:** [January 2, 2025, 6:46pm UTC](https://discuss.elastic.co/t/grok-pattern/372704 "2025-01-02T18:46:48Z")
Hi All, I have below log content which I am capturing from security application so I want to prepare grok pattern to simplify the log however I couldn't make grok pattern for below logs. Kindly suggest what pattern is s… --- ## [Logstash trying to map object on empty text "Illegal argument exception STAT\_OBJECT"](https://discuss.elastic.co/t/logstash-trying-to-map-object-on-empty-text-illegal-argument-exception-stat-object/372622)
**Author:** [@grumo35](https://discuss.elastic.co/u/grumo35)\ **Replies:** 2\ **Last updated:** [December 30, 2024, 2:13pm UTC](https://discuss.elastic.co/t/logstash-trying-to-map-object-on-empty-text-illegal-argument-exception-stat-object/372622 "2024-12-30T14:13:24Z")
Hello, I'm facing an ever lasting issue on some dataset where some "dynamic" plugins like kv or json can map certain field values as object. For example service =\> "{}" I'm not a big fan of per dataset fixes where i t… --- ## [Seeks advice on designing an efficient and reliable Elastic cluster setup, specifically regarding the placement of Logstash and Kibana nodes within a firewall-based architecture](https://discuss.elastic.co/t/seeks-advice-on-designing-an-efficient-and-reliable-elastic-cluster-setup-specifically-regarding-the-placement-of-logstash-and-kibana-nodes-within-a-firewall-based-architecture/372322)
**Author:** [@Weeltin](https://discuss.elastic.co/u/Weeltin)\ **Replies:** 2\ **Last updated:** [December 29, 2024, 9:58am UTC](https://discuss.elastic.co/t/seeks-advice-on-designing-an-efficient-and-reliable-elastic-cluster-setup-specifically-regarding-the-placement-of-logstash-and-kibana-nodes-within-a-firewall-based-architecture/372322 "2024-12-29T09:58:30Z")
Hi everyone, I'm in the process of setting up an Elastic cluster, and I'm looking for some advice on the best way to proceed to ensure I end up with an efficient and reliable solution. Here's my current setup: My i… --- ## [Parsing with logstash shows correct with debug but unable to ingest](https://discuss.elastic.co/t/parsing-with-logstash-shows-correct-with-debug-but-unable-to-ingest/372379)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 2\ **Last updated:** [December 25, 2024, 11:29am UTC](https://discuss.elastic.co/t/parsing-with-logstash-shows-correct-with-debug-but-unable-to-ingest/372379 "2024-12-25T11:29:38Z")
Hi Everyone, I am struggling to ingest messages using logstash, I have been working on the parser for lst 2 weeks but unable to succeed. Grok debugger and rubydebug perfectly show that messages are correctly parsed howe… --- ## [Filebeat doesn't send updated logs](https://discuss.elastic.co/t/filebeat-doesnt-send-updated-logs/372325)
**Author:** [@Sergey\_Ivanov1](https://discuss.elastic.co/u/Sergey_Ivanov1)\ **Replies:** 1\ **Last updated:** [December 24, 2024, 6:13pm UTC](https://discuss.elastic.co/t/filebeat-doesnt-send-updated-logs/372325 "2024-12-24T18:13:03Z")
Good day! Have a problem, my filebeat, wich is installed in Windows doesn't send updated logs. After restart the service, filebeat send all the data from the file, and that is it. After some time I can see that the log … --- ## [Filter part not working as expected for Logstash Filter for processing logs with unique entries](https://discuss.elastic.co/t/filter-part-not-working-as-expected-for-logstash-filter-for-processing-logs-with-unique-entries/372334)
**Author:** [@ErGeek](https://discuss.elastic.co/u/ErGeek)\ **Replies:** 5\ **Last updated:** [December 23, 2024, 7:46pm UTC](https://discuss.elastic.co/t/filter-part-not-working-as-expected-for-logstash-filter-for-processing-logs-with-unique-entries/372334 "2024-12-23T19:46:56Z")
Hi All, We are receiving logs in a Kafka topic in the following format and need to process them using a Logstash filter to send them to Elasticsearch. The requirement is to ignore the first type of log entirely and to s… --- ## [Translate plugin breaking my pipelines](https://discuss.elastic.co/t/translate-plugin-breaking-my-pipelines/372077)
**Author:** [@tammyt](https://discuss.elastic.co/u/tammyt)\ **Replies:** 5\ **Last updated:** [December 23, 2024, 1:22pm UTC](https://discuss.elastic.co/t/translate-plugin-breaking-my-pipelines/372077 "2024-12-23T13:22:14Z")
After updating to 8.16 (and today to 8.17 to see if the issue would be fixed), all pipelines that use translate filter plugin started breaking \[2024-12-17T15:38:27,060\]\[ERROR\]\[logstash.agent \] Failed to execut… --- ## [Decryption of 3rd Party Logs Using Filebeat and Logstash](https://discuss.elastic.co/t/decryption-of-3rd-party-logs-using-filebeat-and-logstash/372302)
**Author:** [@Almog\_Salem](https://discuss.elastic.co/u/Almog_Salem)\ **Replies:** 1\ **Last updated:** [December 22, 2024, 2:17pm UTC](https://discuss.elastic.co/t/decryption-of-3rd-party-logs-using-filebeat-and-logstash/372302 "2024-12-22T14:17:28Z")
Hi, I need some help with configuration and design. I’ve created a Python application that integrates with a specific source and downloads files that do not change. These files are saved on my local Ubuntu machine in a… --- ## [The indexs doesn't shown in kibana](https://discuss.elastic.co/t/the-indexs-doesnt-shown-in-kibana/372239)
**Author:** [@Yasmine\_Ghorbel](https://discuss.elastic.co/u/Yasmine_Ghorbel)\ **Replies:** 1\ **Last updated:** [December 19, 2024, 5:07pm UTC](https://discuss.elastic.co/t/the-indexs-doesnt-shown-in-kibana/372239 "2024-12-19T17:07:04Z")
input { gelf { port =\> 12201 } } filter { mutate { add\_field =\> { "environment" =\> "${ENVIRONMENT}" } add\_tag =\> \[ "%{tag}" \] } if "backend" in \[tags\] { json { … --- ## [Clickhouse](https://discuss.elastic.co/t/clickhouse/372227)
**Author:** [@A\_Mightiev](https://discuss.elastic.co/u/A_Mightiev)\ **Replies:** 0\ **Last updated:** [December 19, 2024, 2:46pm UTC](https://discuss.elastic.co/t/clickhouse/372227 "2024-12-19T14:46:41Z")
Have anyone been able to monitor clickhouse? perhaps an input in logstash to query the state? --- ## [Parsing text and object](https://discuss.elastic.co/t/parsing-text-and-object/372219)
**Author:** [@vahagg1](https://discuss.elastic.co/u/vahagg1)\ **Replies:** 3\ **Last updated:** [December 19, 2024, 11:11am UTC](https://discuss.elastic.co/t/parsing-text-and-object/372219 "2024-12-19T11:11:05Z")
The problem which i face is like this: I have a single field like message which i want to parse it with json parser but sometimes the value which come to it is text and sometimes is json, I try to use regex like "^{.}\[\\… --- ## [Issue with Logstash, ILM and Data stream](https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897)
**Author:** [@fio23](https://discuss.elastic.co/u/fio23)\ **Replies:** 5\ **Last updated:** [December 19, 2024, 7:28am UTC](https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897 "2024-12-19T07:28:34Z")
Hello all, I'm currently trying to create a data stream with a corresponding ILM policy enabled, and then have Logstash to forward logs to that data stream. See below steps I follow: Create Component Template PUT \_co… --- ## [Parsing confluence audit json into Elastic with logstash](https://discuss.elastic.co/t/parsing-confluence-audit-json-into-elastic-with-logstash/371798)
**Author:** [@Logistic\_dilated](https://discuss.elastic.co/u/Logistic_dilated)\ **Replies:** 2\ **Last updated:** [December 19, 2024, 3:02am UTC](https://discuss.elastic.co/t/parsing-confluence-audit-json-into-elastic-with-logstash/371798 "2024-12-19T03:02:26Z")
Hello, I'm testing scraping our company confluence (self hosted) audit api so permission changes etc. are in ELK (also self hosted). I've got an http\_poller pipeline that is successfully polling the api, however the ev… --- ## [Set system time as timestamp](https://discuss.elastic.co/t/set-system-time-as-timestamp/372079)
**Author:** [@Taras\_Mikityuk](https://discuss.elastic.co/u/Taras_Mikityuk)\ **Replies:** 3\ **Last updated:** [December 18, 2024, 12:39pm UTC](https://discuss.elastic.co/t/set-system-time-as-timestamp/372079 "2024-12-18T12:39:55Z")
I'm collecting logs using elastic agent. The problem is, in @timestamp field I have time when event actually happened, but I also need time when log arrived in logstash. I was thinking to use ruby(ruby { code =\> … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=12) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=14)