# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1049 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1050 --- ## [Logstash pipe plugin exec not working](https://discuss.elastic.co/t/logstash-pipe-plugin-exec-not-working/72097)
**Author:** [@Sharat\_Jagannath](https://discuss.elastic.co/u/Sharat_Jagannath)\ **Replies:** 5\ **Last updated:** [January 23, 2017, 9:23pm UTC](https://discuss.elastic.co/t/logstash-pipe-plugin-exec-not-working/72097 "2017-01-23T21:23:51Z")
I have a logstash conf file that gets data from kafka as input and then based on if the input contains a field it is supposed to pipe the input data to a script(unix of python) For example- input{ kafka { zk\_connect =\> … --- ## [Logstash & JSON](https://discuss.elastic.co/t/logstash-json/72377)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 2\ **Last updated:** [January 23, 2017, 5:45pm UTC](https://discuss.elastic.co/t/logstash-json/72377 "2017-01-23T17:45:02Z")
I'm trying to configure Logstash with JSON. logstash configuration: root@596f81c45ea4:~# cat /etc/logstash/conf.d/\* | grep -v \\# input { gelf { codec =\> "json" } } filter { json { source =\> "message" } } … --- ## [Logstash Variables](https://discuss.elastic.co/t/logstash-variables/70822)
**Author:** [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Replies:** 24\ **Last updated:** [January 23, 2017, 3:45pm UTC](https://discuss.elastic.co/t/logstash-variables/70822 "2017-01-23T15:45:59Z")
Hello all, I am doing my level best to understand where I define variable. Please forgive me as I am not very strong in unix as a whole and learning as I go. But this page: https://www.elastic.co/guide/en/logstash/curr… --- ## [Ruby loop](https://discuss.elastic.co/t/ruby-loop/72484)
**Author:** [@Abhamid](https://discuss.elastic.co/u/Abhamid)\ **Replies:** 0\ **Last updated:** [January 23, 2017, 1:48pm UTC](https://discuss.elastic.co/t/ruby-loop/72484 "2017-01-23T13:48:47Z")
Dear, I am a new one in ELK suite, i want use logstash, to update some fields in xml file before insert them in elastcis. My problem is that the xml file, has a nested fields, and from file tof ile, the numebr… --- ## [WARNING: Could not find logstash.yml](https://discuss.elastic.co/t/warning-could-not-find-logstash-yml/71219)
**Author:** [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Replies:** 18\ **Last updated:** [January 23, 2017, 1:44pm UTC](https://discuss.elastic.co/t/warning-could-not-find-logstash-yml/71219 "2017-01-23T13:44:45Z")
Hello/Bonjour I have installed ELK. Note : All lines of differents config files not mentionned after are commented (#xyabc) Elasticsearch conf in "/etc/elasticsearch/elasticsearch.yml" is : node.name: elastic1 … --- ## [Warnings after changing logstash 5.1.1 to 5.1.2](https://discuss.elastic.co/t/warnings-after-changing-logstash-5-1-1-to-5-1-2/72030)
**Author:** [@tbw](https://discuss.elastic.co/u/tbw)\ **Replies:** 5\ **Last updated:** [January 23, 2017, 1:43pm UTC](https://discuss.elastic.co/t/warnings-after-changing-logstash-5-1-1-to-5-1-2/72030 "2017-01-23T13:43:29Z")
After updating from logstash 5.1.1 to 5.1.2 (also elastic) i see those messages: {"log":"14:56:02.635 \[\[main\]\\u003eworker0\] WARN logstash.outputs.elasticsearch - Failed action. {:status=\\u003e409, :action=\\u003e\["upda… --- ## [Logstash validate fields against ES mappings](https://discuss.elastic.co/t/logstash-validate-fields-against-es-mappings/72475)
**Author:** [@nick-george](https://discuss.elastic.co/u/nick-george)\ **Replies:** 0\ **Last updated:** [January 23, 2017, 11:45am UTC](https://discuss.elastic.co/t/logstash-validate-fields-against-es-mappings/72475 "2017-01-23T11:45:26Z")
Hi LS community, Are you aware of a filter that can compare the list of fields in an event against those in a ES mapping template (JSON file)? Ideally, the filter would then "isolate" those fields that are found in th… --- ## [Input JSON file to elasticsearch via logstash](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437)
**Author:** [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\ **Replies:** 13\ **Last updated:** [January 23, 2017, 10:01am UTC](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437 "2017-01-23T10:01:42Z")
hi, im facing an issue with logstash while inserting json data to elasticsearch. here is my logstash config file input { file { codec =\> multiline { pattern =\> '^{' negate =\> true what =\> previous } pa… --- ## [NumberFormatException stops logstash from working](https://discuss.elastic.co/t/numberformatexception-stops-logstash-from-working/71973)
**Author:** [@guenther](https://discuss.elastic.co/u/guenther)\ **Replies:** 14\ **Last updated:** [January 23, 2017, 8:04am UTC](https://discuss.elastic.co/t/numberformatexception-stops-logstash-from-working/71973 "2017-01-23T08:04:38Z")
Hi, We have logstash running for a while now, but yesterday something happened that stopped it from working. \[2017-01-18T10:56:16,553\]\[DEBUG\]\[logstash.inputs.beats \] config LogStash::Inputs::Beats/@client\_inacti… --- ## [Geoip "illegal latitude" after upgrading 1.5 -\> 2.1](https://discuss.elastic.co/t/geoip-illegal-latitude-after-upgrading-1-5-2-1/40083)
**Author:** [@ceekay](https://discuss.elastic.co/u/ceekay)\ **Replies:** 4\ **Last updated:** [January 23, 2017, 3:53am UTC](https://discuss.elastic.co/t/geoip-illegal-latitude-after-upgrading-1-5-2-1/40083 "2017-01-23T03:53:32Z")
I've just upgraded some boxes from 1.5 to 2.1 and have been getting indexing errors for all messages containing geoip data ever since. Messages that don't contain geoip info are indexing without errors. As far as I c… --- ## [Location of logstash.yml using brew?](https://discuss.elastic.co/t/location-of-logstash-yml-using-brew/72398)
**Author:** [@scheung38](https://discuss.elastic.co/u/scheung38)\ **Replies:** 1\ **Last updated:** [January 22, 2017, 7:53pm UTC](https://discuss.elastic.co/t/location-of-logstash-yml-using-brew/72398 "2017-01-22T19:53:50Z")
Hi I am a new user to logstash but could not work out where to put logstash.yml? $brew install logstash Am I supposed to generate this file? --- ## [What's the best way to add some additional information from Logstash to Elasticsearch](https://discuss.elastic.co/t/whats-the-best-way-to-add-some-additional-information-from-logstash-to-elasticsearch/72396)
**Author:** [@CaptainFeng](https://discuss.elastic.co/u/CaptainFeng)\ **Replies:** 1\ **Last updated:** [January 22, 2017, 3:01pm UTC](https://discuss.elastic.co/t/whats-the-best-way-to-add-some-additional-information-from-logstash-to-elasticsearch/72396 "2017-01-22T15:01:50Z")
Hi All I am a Newbie in using Logstash and Elasticsearch. Recently, I got a job which parse some logs and put them into Elasticsearch. And before putting them in, I need to using information of every line to retrieve … --- ## [Rename Value to Friendly readable name](https://discuss.elastic.co/t/rename-value-to-friendly-readable-name/72361)
**Author:** [@horwedja](https://discuss.elastic.co/u/horwedja)\ **Replies:** 2\ **Last updated:** [January 22, 2017, 8:57am UTC](https://discuss.elastic.co/t/rename-value-to-friendly-readable-name/72361 "2017-01-22T08:57:54Z")
Hi, I am new to Logstash. i ship multiple firewall logs to logstash and we identify each firewall by SN. The SN is not easily identified and want to give each SN a friendly name. I have tried to use mutate rename{} but t… --- ## [S3 Input - No Time Information Error](https://discuss.elastic.co/t/s3-input-no-time-information-error/45150)
**Author:** [@geoffdinsdale](https://discuss.elastic.co/u/geoffdinsdale)\ **Replies:** 1\ **Last updated:** [January 22, 2017, 5:50am UTC](https://discuss.elastic.co/t/s3-input-no-time-information-error/45150 "2017-01-22T05:50:07Z")
Getting an issue when trying to import log files from an S3 bucket to ES, be assured that I do have the credentials to hit the bucket and can in fact copy the files and process them with a file input, but this limits our… --- ## [Routing events to specific workers](https://discuss.elastic.co/t/routing-events-to-specific-workers/72363)
**Author:** [@val](https://discuss.elastic.co/u/val)\ **Replies:** 2\ **Last updated:** [January 22, 2017, 5:15am UTC](https://discuss.elastic.co/t/routing-events-to-specific-workers/72363 "2017-01-22T05:15:17Z")
In case Logstash runs with multiple pipeline workers, events are handed out to each worker in turn as they get available. Certain plugins (e.g. such as aggregate but also a few others doing some in-memory caching) requir… --- ## [Special Character as URL Params](https://discuss.elastic.co/t/special-character-as-url-params/72093)
**Author:** [@sbruyere](https://discuss.elastic.co/u/sbruyere)\ **Replies:** 6\ **Last updated:** [January 21, 2017, 10:42pm UTC](https://discuss.elastic.co/t/special-character-as-url-params/72093 "2017-01-21T22:42:02Z")
Hello, I try tu use the following URL in a Http\_poller Input Plugin. url =\> "http://10.0.10.10/login/login.cgi?mode=AUTH\_LOGIN¶ms=userid|password&" But the special character " | " (between userid and password as… --- ## [Kibana Timezone settings](https://discuss.elastic.co/t/kibana-timezone-settings/71793)
**Author:** [@sharon.c](https://discuss.elastic.co/u/sharon.c)\ **Replies:** 4\ **Last updated:** [January 19, 2017, 1:21am UTC](https://discuss.elastic.co/t/kibana-timezone-settings/71793 "2017-01-19T01:21:43Z")
I am using logstash 5.1 streaming mysql data to elasticsearch 5.1, kibana 5 and this is my logstash code: input { jdbc { jdbc\_driver\_library =\> "./mysql-connector-java-5.1.36.jar" jdbc\_driver\_class =\> "com.m… --- ## [Amazon\_es output not signing?](https://discuss.elastic.co/t/amazon-es-output-not-signing/72356)
**Author:** [@mressler](https://discuss.elastic.co/u/mressler)\ **Replies:** 2\ **Last updated:** [January 21, 2017, 2:12pm UTC](https://discuss.elastic.co/t/amazon-es-output-not-signing/72356 "2017-01-21T14:12:28Z")
Hi Logstash experts, I've configured my logstash to use the amazon\_es output plugin and everything seems to be fine on configtest. I have region, aws\_access\_key\_id, and aws\_secret\_access\_key set correctly - I even star… --- ## [The differences of logstash path: logstash.log and logstash-plain.log](https://discuss.elastic.co/t/the-differences-of-logstash-path-logstash-log-and-logstash-plain-log/72354)
**Author:** [@nanshan](https://discuss.elastic.co/u/nanshan)\ **Replies:** 0\ **Last updated:** [January 20, 2017, 11:52pm UTC](https://discuss.elastic.co/t/the-differences-of-logstash-path-logstash-log-and-logstash-plain-log/72354 "2017-01-20T23:52:10Z")
Logstash version: 5.1.2 total 1016004 -rw-r--r-- 1 logstash root 0 Jan 20 04:51 logstash.err -rw-r--r-- 1 logstash logstash 23260984 Jan 20 22:53 logstash.log -rw-r--r-- 1 logstash logstash 1016955863 J… --- ## [Logstash metric not displaying %{count} in the output](https://discuss.elastic.co/t/logstash-metric-not-displaying-count-in-the-output/72347)
**Author:** [@irom77](https://discuss.elastic.co/u/irom77)\ **Replies:** 1\ **Last updated:** [January 20, 2017, 11:18pm UTC](https://discuss.elastic.co/t/logstash-metric-not-displaying-count-in-the-output/72347 "2017-01-20T23:18:00Z")
I have metrics adding 'Counted" tag and sending email when count is greater than 1. All is fine except I receive THREAT count: %{count} 0.20536684761303686 in subject instead of THREAT count: 2 0.20536684761303686 if "… --- ## [Logstash-output-http retry not working](https://discuss.elastic.co/t/logstash-output-http-retry-not-working/72343)
**Author:** [@saching](https://discuss.elastic.co/u/saching)\ **Replies:** 0\ **Last updated:** [January 20, 2017, 9:26pm UTC](https://discuss.elastic.co/t/logstash-output-http-retry-not-working/72343 "2017-01-20T21:26:22Z")
output { http { url =\> "http://localhost:8080/MyApp/api/test" http\_method =\> "post" connect\_timeout =\> 30 request\_timeout =\> 30 socket\_timeout =\> 30 automatic\_retries =\> 3 } } It fails with … --- ## [Logstash Rabbitmq IO exception](https://discuss.elastic.co/t/logstash-rabbitmq-io-exception/72331)
**Author:** [@piyush](https://discuss.elastic.co/u/piyush)\ **Replies:** 0\ **Last updated:** [January 20, 2017, 8:20pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-io-exception/72331 "2017-01-20T20:20:31Z")
Getting below exception while trying to pull logs from Rabbitmq, how can i debug more? Connecting to RabbitMQ. Settings: {:vhost=\>"/", :host=\>"1.1.1.1", :port=\>5671, :user=\>"a", :automatic\_recovery=\>true, :pass=\>"a", :… --- ## [Logstash 5.0 logstash-kafka-input issues](https://discuss.elastic.co/t/logstash-5-0-logstash-kafka-input-issues/72321)
**Author:** [@Maxwell\_Flanders](https://discuss.elastic.co/u/Maxwell_Flanders)\ **Replies:** 0\ **Last updated:** [January 20, 2017, 6:32pm UTC](https://discuss.elastic.co/t/logstash-5-0-logstash-kafka-input-issues/72321 "2017-01-20T18:32:38Z")
I have been using the logstash-kafka-input plugin for a while now, on logstash version 2.1.0. We recently just upgraded out logstash instances from 2.1.0 to 5.1.2, and we have started seeing errors of this type rather fr… --- ## [Logstash, Netflow Codec, Template length doesn't fit cleanly](https://discuss.elastic.co/t/logstash-netflow-codec-template-length-doesnt-fit-cleanly/72320)
**Author:** [@ave19](https://discuss.elastic.co/u/ave19)\ **Replies:** 0\ **Last updated:** [January 20, 2017, 6:28pm UTC](https://discuss.elastic.co/t/logstash-netflow-codec-template-length-doesnt-fit-cleanly/72320 "2017-01-20T18:28:00Z")
I am not a netflow expert. I am not a codec expert. Really, not much of an expert in anything, just getting that out of the way. I am receiving netflow events, most seem to be coming through OK. I'm not sure what to… --- ## [Failed to submit a listener notification task... A plugin had an unrecoverable error](https://discuss.elastic.co/t/failed-to-submit-a-listener-notification-task-a-plugin-had-an-unrecoverable-error/60932)
**Author:** [@agonzalez](https://discuss.elastic.co/u/agonzalez)\ **Replies:** 8\ **Last updated:** [January 20, 2017, 6:06pm UTC](https://discuss.elastic.co/t/failed-to-submit-a-listener-notification-task-a-plugin-had-an-unrecoverable-error/60932 "2017-01-20T18:06:22Z")
I am seeing this error in logstash every few seconds. any help how to fix this? I am running 5.0-alpha5. 20:34:34.228 \[\[main\]\ **Author:** [@inigo](https://discuss.elastic.co/u/inigo)\ **Replies:** 0\ **Last updated:** [January 20, 2017, 6:05pm UTC](https://discuss.elastic.co/t/logstash-output-lumberjack-plugin-not-packaged-with-logstash-5-1/72316 "2017-01-20T18:05:15Z") According to the documentation, the lumberjack output plugin should be available with Logstash 5.1, however I'm seeing the following error when trying to use it: \[2017-01-20T14:47:27,608\]\[ERROR\]\[logstash.plugins.regist… --- ## [Collecting from multiple sources](https://discuss.elastic.co/t/collecting-from-multiple-sources/72070)
**Author:** [@mmethe](https://discuss.elastic.co/u/mmethe)\ **Replies:** 7\ **Last updated:** [January 20, 2017, 5:56pm UTC](https://discuss.elastic.co/t/collecting-from-multiple-sources/72070 "2017-01-20T17:56:47Z")
Hello, Kind of new to all this so hang in there. ELK running on Centos 7 Winlogbeat sending logs from several windows servers to ELK When I try to index logstash (rsyslog), I dont see any data. here is the inpu… --- ## [Ruby filter loop through fields in logstash 5?](https://discuss.elastic.co/t/ruby-filter-loop-through-fields-in-logstash-5/72026)
**Author:** [@Maxwell\_Flanders](https://discuss.elastic.co/u/Maxwell_Flanders)\ **Replies:** 2\ **Last updated:** [January 20, 2017, 12:53pm UTC](https://discuss.elastic.co/t/ruby-filter-loop-through-fields-in-logstash-5/72026 "2017-01-20T12:53:36Z")
In logstash 2.1 I have been using a ruby filter to loop through all top-level fields and do a gsub to remove some special characters. I know that in 5.0, the Event API has changed significantly with an impact on ruby fil… --- ## [Changing time zone format](https://discuss.elastic.co/t/changing-time-zone-format/71775)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 19\ **Last updated:** [January 20, 2017, 11:52am UTC](https://discuss.elastic.co/t/changing-time-zone-format/71775 "2017-01-20T11:52:02Z")
i am using jdbc as input and http as output. from jdbc i am getting date as "2017-01-02T12:35:44.000Z" but in my http output date will be accept as "2017-01-02T12:35:44.000+07:00" when i am checking inn database the… --- ## [Different lines structure for same log](https://discuss.elastic.co/t/different-lines-structure-for-same-log/71779)
**Author:** [@przecie](https://discuss.elastic.co/u/przecie)\ **Replies:** 4\ **Last updated:** [January 20, 2017, 9:27am UTC](https://discuss.elastic.co/t/different-lines-structure-for-same-log/71779 "2017-01-20T09:27:50Z")
Hi I have a following lines in my log: -- - 2017-01-16 00:00:01,779 \[Thread-31:SpaceChangePollerTask(SpaceChangePollerTask-1121721-Space:MAIN)\] INFO (support.context.FrameworkSession) Using overridden system date:… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1048) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1050)