# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=656 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 657 --- ## [How to catch an error in logstash](https://discuss.elastic.co/t/how-to-catch-an-error-in-logstash/154499)
**Author:** [@zahodi](https://discuss.elastic.co/u/zahodi)\ **Replies:** 2\ **Last updated:** [November 1, 2018, 4:20am UTC](https://discuss.elastic.co/t/how-to-catch-an-error-in-logstash/154499 "2018-11-01T04:20:28Z")
Hello, my logstash json filter seems be throwing errors on certain messages and I'm nut sure how to debug it. Is there a way to figure what messages or what is source of the message that threw this exception. I have mul… --- ## [Error trying to run rspec on a custom logstash filter](https://discuss.elastic.co/t/error-trying-to-run-rspec-on-a-custom-logstash-filter/154940)
**Author:** [@pabloaredu](https://discuss.elastic.co/u/pabloaredu)\ **Replies:** 1\ **Last updated:** [October 31, 2018, 11:46pm UTC](https://discuss.elastic.co/t/error-trying-to-run-rspec-on-a-custom-logstash-filter/154940 "2018-10-31T23:46:39Z")
Hi all, Does anyone have any experience with running rspec on Logstash filters for Logstash 6\*?? I've tried to run bundle exec rspec but keep getting this error: no such file to load -- org/apache/logging/log4j/log4j-c… --- ## [How to Remove deleted data from index data](https://discuss.elastic.co/t/how-to-remove-deleted-data-from-index-data/154919)
**Author:** [@Dipak\_Shelke](https://discuss.elastic.co/u/Dipak_Shelke)\ **Replies:** 0\ **Last updated:** [October 31, 2018, 7:33pm UTC](https://discuss.elastic.co/t/how-to-remove-deleted-data-from-index-data/154919 "2018-10-31T19:33:19Z")
Hi I have an input file having 10 records which is already indexed. Now out of 10 records 4 records removed from file and 6 records are updated. I have used document\_id to replace all the records in that index. Now Th… --- ## [How to add a for loop and if condition in logstash input plugin](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888)
**Author:** [@Bharath\_Pusuluri](https://discuss.elastic.co/u/Bharath_Pusuluri)\ **Replies:** 9\ **Last updated:** [October 31, 2018, 6:02pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888 "2018-10-31T18:02:25Z")
My requirement is sa follows: I am running the following python script in input field. The Python script contains a curl command and outputting that data as input (JSON format). input { exec { type =\> "apps" command… --- ## [Looking for a way to prune fields with certain text in them but struggling with the prune filter and regex](https://discuss.elastic.co/t/looking-for-a-way-to-prune-fields-with-certain-text-in-them-but-struggling-with-the-prune-filter-and-regex/154804)
**Author:** [@shog1466112](https://discuss.elastic.co/u/shog1466112)\ **Replies:** 1\ **Last updated:** [October 31, 2018, 5:57pm UTC](https://discuss.elastic.co/t/looking-for-a-way-to-prune-fields-with-certain-text-in-them-but-struggling-with-the-prune-filter-and-regex/154804 "2018-10-31T17:57:17Z")
I'm pushing logstash node stats into Elasticsearch with the following pipeline setup: input { exec { command =\> "curl -s http://localhost:9600/\_node/stats" interval =\> 15 } } filter { json { source =\> … --- ## [Logstash-s3-output plugin not working without s3:deleteObject permission](https://discuss.elastic.co/t/logstash-s3-output-plugin-not-working-without-s3-deleteobject-permission/154875)
**Author:** [@Arnold\_Kell](https://discuss.elastic.co/u/Arnold_Kell)\ **Replies:** 1\ **Last updated:** [October 31, 2018, 4:38pm UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin-not-working-without-s3-deleteobject-permission/154875 "2018-10-31T16:38:17Z")
Logstash output would not work for me without s3:deleteObject permission although as I understand, only the s3:putObject permission is required for the plugin to work properly. Without the s3:deleteObject an xml file app… --- ## [Logstash is not reading logstash.conf file](https://discuss.elastic.co/t/logstash-is-not-reading-logstash-conf-file/152954)
**Author:** [@krajapraveen](https://discuss.elastic.co/u/krajapraveen)\ **Replies:** 2\ **Last updated:** [October 31, 2018, 3:54pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-logstash-conf-file/152954 "2018-10-31T15:54:47Z")
Hi I have installed Elastic Search, Logstash and Kibana on my windows laptop. Windows directory path to Kibana: C:\\Users\\rajapraveenk\\Downloads\\kibana-6.4.2-windows-x86\_64 Windows directory path to Logstash: C:\\Users… --- ## [Unable to start logstash after the setup](https://discuss.elastic.co/t/unable-to-start-logstash-after-the-setup/154828)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 1\ **Last updated:** [October 31, 2018, 3:12pm UTC](https://discuss.elastic.co/t/unable-to-start-logstash-after-the-setup/154828 "2018-10-31T15:12:36Z")
HI Team, I am getting below error while starting the logstash with a sample conf file present in default conf.d folder. \[ERROR\] 2018-10-31 04:45:22.819 \[Converge PipelineAction::Create\\] agent - Failed to execute … --- ## [Index is loaded with data from other indexes](https://discuss.elastic.co/t/index-is-loaded-with-data-from-other-indexes/154859)
**Author:** [@bueka.torao](https://discuss.elastic.co/u/bueka.torao)\ **Replies:** 0\ **Last updated:** [October 31, 2018, 2:19pm UTC](https://discuss.elastic.co/t/index-is-loaded-with-data-from-other-indexes/154859 "2018-10-31T14:19:41Z")
Hi all, I need some help here! I am reading logs from my FTP server, also from some different servers I am using filebeats + logstash + elasticsearch + kibana. Mail server -\> filebeats -\> logstash 5045 index maillog-\* … --- ## [Handling duplicates from SQL Dumps/json and API response in Logstash](https://discuss.elastic.co/t/handling-duplicates-from-sql-dumps-json-and-api-response-in-logstash/154351)
**Author:** [@wheelq](https://discuss.elastic.co/u/wheelq)\ **Replies:** 3\ **Last updated:** [October 31, 2018, 1:51pm UTC](https://discuss.elastic.co/t/handling-duplicates-from-sql-dumps-json-and-api-response-in-logstash/154351 "2018-10-31T13:51:57Z")
I am just about to start importing SQL Dumps and API responses from one of the systems. But I have realised that those data dumps and API responses will contain same information all the time, but also: some of the data… --- ## [How to use send query instead of elastic to logstash?](https://discuss.elastic.co/t/how-to-use-send-query-instead-of-elastic-to-logstash/154842)
**Author:** [@mhsankar](https://discuss.elastic.co/u/mhsankar)\ **Replies:** 7\ **Last updated:** [October 31, 2018, 1:02pm UTC](https://discuss.elastic.co/t/how-to-use-send-query-instead-of-elastic-to-logstash/154842 "2018-10-31T13:02:28Z")
Hi How can send my query instead of elastic to logstash? i want to use logstash to get my query from elastic. how can i use it? --- ## [ElasticSearch No Available connections error in Logstash](https://discuss.elastic.co/t/elasticsearch-no-available-connections-error-in-logstash/154581)
**Author:** [@marpa](https://discuss.elastic.co/u/marpa)\ **Replies:** 1\ **Last updated:** [October 31, 2018, 12:56pm UTC](https://discuss.elastic.co/t/elasticsearch-no-available-connections-error-in-logstash/154581 "2018-10-31T12:56:24Z")
I'm running a Logstash instance which is connected to an ES cluster behind a load balancer. The load balancer has an idle timeout of 5 minutes. Logstash is configured with the ES url corresponding to the loadbalancer i… --- ## [Logstash Opening file only on quit](https://discuss.elastic.co/t/logstash-opening-file-only-on-quit/154234)
**Author:** [@wheelq](https://discuss.elastic.co/u/wheelq)\ **Replies:** 1\ **Last updated:** [October 31, 2018, 12:52pm UTC](https://discuss.elastic.co/t/logstash-opening-file-only-on-quit/154234 "2018-10-31T12:52:56Z")
Very strange thing is happening. Logstash is being launched with the following command: logstash -f /tmp/my\_config.yml sincedb\_path -\> set to /dev/null Input file is being generated- nothing happens. Input file is b… --- ## [The shutdown process appears to be stalled due to busy or blocked plugins](https://discuss.elastic.co/t/the-shutdown-process-appears-to-be-stalled-due-to-busy-or-blocked-plugins/154257)
**Author:** [@wheelq](https://discuss.elastic.co/u/wheelq)\ **Replies:** 1\ **Last updated:** [October 31, 2018, 12:52pm UTC](https://discuss.elastic.co/t/the-shutdown-process-appears-to-be-stalled-due-to-busy-or-blocked-plugins/154257 "2018-10-31T12:52:15Z")
I have hit same issue: https://discuss.elastic.co/t/the-shutdown-process-appears-to-be-stalled-due-to-busy-or-blocked-plugins/135000 No erros in the logs... --- ## [How to use metadata id to condition check of Logstash output which use jdbc input?](https://discuss.elastic.co/t/how-to-use-metadata-id-to-condition-check-of-logstash-output-which-use-jdbc-input/154821)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 0\ **Last updated:** [October 31, 2018, 11:11am UTC](https://discuss.elastic.co/t/how-to-use-metadata-id-to-condition-check-of-logstash-output-which-use-jdbc-input/154821 "2018-10-31T11:11:50Z")
Hi, I am using ELK GA 6.3.0. I have a logstash input like; input { jdbc { id =\> "jdbc1" . . . statement =\> "some sql statement" add\_field =\> { "logtype" =\> "jdbc\_log\_1" } } } and I have multiple jdbc inp… --- ## [Working with large files 100gb+](https://discuss.elastic.co/t/working-with-large-files-100gb/154779)
**Author:** [@mrncgirc](https://discuss.elastic.co/u/mrncgirc)\ **Replies:** 1\ **Last updated:** [October 31, 2018, 10:31am UTC](https://discuss.elastic.co/t/working-with-large-files-100gb/154779 "2018-10-31T10:31:34Z")
Hi, Related to my recent threads regarding XML Import into Logstash. I now have my entire import set up, and it runs as expected in the test environment. However, working with production files I am running into an issue… --- ## [\[Filter ruby plugin\] How to add exception message to event?](https://discuss.elastic.co/t/filter-ruby-plugin-how-to-add-exception-message-to-event/154813)
**Author:** [@Sergey34](https://discuss.elastic.co/u/Sergey34)\ **Replies:** 0\ **Last updated:** [October 31, 2018, 10:31am UTC](https://discuss.elastic.co/t/filter-ruby-plugin-how-to-add-exception-message-to-event/154813 "2018-10-31T10:31:22Z")
Exceptions may be thrown when working with ruby filter. Now the event logs only the fact that there was an exception, but it is not informative. is it possible to add exception message to tags? I propose to add the ab… --- ## [Not able to load text files into elastic search](https://discuss.elastic.co/t/not-able-to-load-text-files-into-elastic-search/153220)
**Author:** [@ruban7483](https://discuss.elastic.co/u/ruban7483)\ **Replies:** 8\ **Last updated:** [October 31, 2018, 10:17am UTC](https://discuss.elastic.co/t/not-able-to-load-text-files-into-elastic-search/153220 "2018-10-31T10:17:19Z")
I am new to elastic search. I have a code to load text files into elastic search. The text file has two columns and its not working. But the same code is working fine for single column. Please some one help me to fix thi… --- ## [I try to input log with filebead to logstash . but can not do it](https://discuss.elastic.co/t/i-try-to-input-log-with-filebead-to-logstash-but-can-not-do-it/154802)
**Author:** [@bgt](https://discuss.elastic.co/u/bgt)\ **Replies:** 0\ **Last updated:** [October 31, 2018, 10:05am UTC](https://discuss.elastic.co/t/i-try-to-input-log-with-filebead-to-logstash-but-can-not-do-it/154802 "2018-10-31T10:05:20Z")
hi everybody im new elk user . i can use filebeat input log to elasticsearch and see log with kibana but i try to use filebeat input log to logstash and putout to elasticsearch its not work. i need help ... this is m… --- ## [\[Input jdbc plugin\] Incorrect value on "last\_run\_metadata\_path" file](https://discuss.elastic.co/t/input-jdbc-plugin-incorrect-value-on-last-run-metadata-path-file/154788)
**Author:** [@vara0855](https://discuss.elastic.co/u/vara0855)\ **Replies:** 0\ **Last updated:** [October 31, 2018, 9:28am UTC](https://discuss.elastic.co/t/input-jdbc-plugin-incorrect-value-on-last-run-metadata-path-file/154788 "2018-10-31T09:28:16Z")
Logstash - v.6.3.2 OSS ES - v.6.3.2 OSS Oracle DB Example input: jdbc { id =\> "input\_jdbc" jdbc\_driver\_library =\> "${LOGSTASH\_JDBC\_DRIVER:ojdbc6.jar}" jdbc\_driver\_class =\> "Java::oracle.jdbc.OracleDriver" jd… --- ## [Trouble for grab events with aggregate filter](https://discuss.elastic.co/t/trouble-for-grab-events-with-aggregate-filter/154214)
**Author:** [@hummermania](https://discuss.elastic.co/u/hummermania)\ **Replies:** 1\ **Last updated:** [October 31, 2018, 8:56am UTC](https://discuss.elastic.co/t/trouble-for-grab-events-with-aggregate-filter/154214 "2018-10-31T08:56:45Z")
Hello experts! I try to parse log which have field used for task\_id, some other lines with same format - event details, and also have end marker for completelly ending event. For example: filter { grok { match =\> { … --- ## [Logstash output rabbitmq](https://discuss.elastic.co/t/logstash-output-rabbitmq/154401)
**Author:** [@flochon](https://discuss.elastic.co/u/flochon)\ **Replies:** 3\ **Last updated:** [October 31, 2018, 8:48am UTC](https://discuss.elastic.co/t/logstash-output-rabbitmq/154401 "2018-10-31T08:48:34Z")
Hello, I use the output of logstash to connect to Rabbitmq with ssl. I have this config for Logstash : output { rabbitmq { host =\> "rabbitmq" port =\> 5671 exchange =\> "logstash" exchange\_type =\> "dir… --- ## [JMS Integration](https://discuss.elastic.co/t/jms-integration/154753)
**Author:** [@Ajit\_Gadge](https://discuss.elastic.co/u/Ajit_Gadge)\ **Replies:** 0\ **Last updated:** [October 31, 2018, 6:27am UTC](https://discuss.elastic.co/t/jms-integration/154753 "2018-10-31T06:27:27Z")
Hi , We are trying to integrate JMS ( Java Messaging Queue ) with logstash. We know that there is plugin for JMS integration with logstash (https://www.elastic.co/guide/en/logstash-versioned-plugins/versioned\_plugin\_doc… --- ## [Multiple pipelines with distributor pattern](https://discuss.elastic.co/t/multiple-pipelines-with-distributor-pattern/152210)
**Author:** [@Kamen\_MacKay](https://discuss.elastic.co/u/Kamen_MacKay)\ **Replies:** 2\ **Last updated:** [October 30, 2018, 4:46pm UTC](https://discuss.elastic.co/t/multiple-pipelines-with-distributor-pattern/152210 "2018-10-30T16:46:25Z")
I've just started working with multiple pipelines and the distributor pattern. Am I correct in my understanding that I have to copy'n'paste my individual logstash config contents for each pipeline into pipelines.yml in o… --- ## [Logstash java related errors](https://discuss.elastic.co/t/logstash-java-related-errors/154523)
**Author:** [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\ **Replies:** 3\ **Last updated:** [October 30, 2018, 10:27pm UTC](https://discuss.elastic.co/t/logstash-java-related-errors/154523 "2018-10-30T22:27:05Z")
Hello there, All the sudden, our developers reported that logstash started working abnormally. I took a look on the all nodes of ELK, kibana and elastisticsearch nodes are working fine. I'm able to log on to kibana an… --- ## [Nested Json array extraction using logstash](https://discuss.elastic.co/t/nested-json-array-extraction-using-logstash/154689)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 0\ **Last updated:** [October 30, 2018, 4:06pm UTC](https://discuss.elastic.co/t/nested-json-array-extraction-using-logstash/154689 "2018-10-30T16:06:39Z")
HI, I working some tricky log and it contain nested array with json formate. Each array contain same fieldname. I want to show those fieldname and count on vertical graph Single event {"bridgeNemoDataType":"rally-test… --- ## [Logstash: Weblogic: Filebeat : Issues while starting the Logstash](https://discuss.elastic.co/t/logstash-weblogic-filebeat-issues-while-starting-the-logstash/154041)
**Author:** [@Shashidhar\_Wl](https://discuss.elastic.co/u/Shashidhar_Wl)\ **Replies:** 4\ **Last updated:** [October 30, 2018, 5:55pm UTC](https://discuss.elastic.co/t/logstash-weblogic-filebeat-issues-while-starting-the-logstash/154041 "2018-10-30T17:55:11Z")
Hi, I am trying to configure ELK with Weblogic logs. where i am getting the plug in registering issues. when i do config test it returns configuration looks good but when i start it is not starting. ./logstash --config… --- ## [How can I identify backlog in pipeline?](https://discuss.elastic.co/t/how-can-i-identify-backlog-in-pipeline/154697)
**Author:** [@Darren\_Stokes](https://discuss.elastic.co/u/Darren_Stokes)\ **Replies:** 0\ **Last updated:** [October 30, 2018, 4:40pm UTC](https://discuss.elastic.co/t/how-can-i-identify-backlog-in-pipeline/154697 "2018-10-30T16:40:44Z")
I am using logspout to send logs to logstash and then onto elasticsearch. Occassionally I'm seeing some logs appear more than half an hour after they were produced. Could someone help me understand where I can look to … --- ## [Issues with XML](https://discuss.elastic.co/t/issues-with-xml/154457)
**Author:** [@Vkryvor](https://discuss.elastic.co/u/Vkryvor)\ **Replies:** 1\ **Last updated:** [October 30, 2018, 3:29pm UTC](https://discuss.elastic.co/t/issues-with-xml/154457 "2018-10-30T15:29:52Z")
Hey all, I'm trying to pull information from a XML file. I've been reading about the xml filter and its abilities for the last week or so and it's still got me stumped. Here's my config file and the sample xml file I'm … --- ## [Logstash does not stop after error connect to database](https://discuss.elastic.co/t/logstash-does-not-stop-after-error-connect-to-database/153403)
**Author:** [@Jakub500](https://discuss.elastic.co/u/Jakub500)\ **Replies:** 1\ **Last updated:** [October 30, 2018, 2:46pm UTC](https://discuss.elastic.co/t/logstash-does-not-stop-after-error-connect-to-database/153403 "2018-10-30T14:46:12Z")
I have a problem with Logstash. The problem is that Logstash does not stop working, when it has problem with connect to database. I use a input plugin and i have correct configuration (according to documentation), I mean… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=655) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=657)