# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1132 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1133 --- ## [How to add geo\_point type data to elasticsearch?](https://discuss.elastic.co/t/how-to-add-geo-point-type-data-to-elasticsearch/2768)
**Author:** [@Eric](https://discuss.elastic.co/u/Eric)\ **Replies:** 5\ **Last updated:** [July 4, 2016, 9:22pm UTC](https://discuss.elastic.co/t/how-to-add-geo-point-type-data-to-elasticsearch/2768 "2016-07-04T21:22:56Z")
Hi, I'm a newbie for ELK. I want to add some custom geo search functions to my program. How do I filter custom lat and lng (now i have %{lat} and %{lng}) into elasticsearch geo\_type formated data so that I can visualize … --- ## [Add\_field not performing variable substitution](https://discuss.elastic.co/t/add-field-not-performing-variable-substitution/53629)
**Author:** [@jeffsmartpipe](https://discuss.elastic.co/u/jeffsmartpipe)\ **Replies:** 1\ **Last updated:** [July 4, 2016, 6:48pm UTC](https://discuss.elastic.co/t/add-field-not-performing-variable-substitution/53629 "2016-07-04T18:48:45Z")
Hi all, Newbie to logstash here but I'm sure this should work. I am creating a metric and need to be able to split by http result code. The result code is stored in %{rcode}. In the below the variable is being substitu… --- ## [Not able to parse http version from weblogic access log in logstash configuration](https://discuss.elastic.co/t/not-able-to-parse-http-version-from-weblogic-access-log-in-logstash-configuration/53639)
**Author:** [@kunalp](https://discuss.elastic.co/u/kunalp)\ **Replies:** 1\ **Last updated:** [July 4, 2016, 6:47pm UTC](https://discuss.elastic.co/t/not-able-to-parse-http-version-from-weblogic-access-log-in-logstash-configuration/53639 "2016-07-04T18:47:06Z")
Here is my configuration file. input{ file{ path =\> "/home/kunal/Desktop/log/accessWeblogic.log" start\_position =\> beginning ignore\_older =\> 0 } } f… --- ## [Clustering Logstash shippers](https://discuss.elastic.co/t/clustering-logstash-shippers/53640)
**Author:** [@Adir](https://discuss.elastic.co/u/Adir)\ **Replies:** 2\ **Last updated:** [July 4, 2016, 6:43pm UTC](https://discuss.elastic.co/t/clustering-logstash-shippers/53640 "2016-07-04T18:43:46Z")
Hi, We are preparing our environment to Logstash deployment and i have one question about clustering of Logstash shippers: On all the documents i have read about clustering Logstash i saw that we should have 2 or more s… --- ## [Parsing XML files with Logstash in Windows](https://discuss.elastic.co/t/parsing-xml-files-with-logstash-in-windows/54473)
**Author:** [@Jemli\_Fathi](https://discuss.elastic.co/u/Jemli_Fathi)\ **Replies:** 5\ **Last updated:** [July 4, 2016, 6:41pm UTC](https://discuss.elastic.co/t/parsing-xml-files-with-logstash-in-windows/54473 "2016-07-04T18:41:45Z")
Hi, I m trying to parse XML files in Windows under C:\\busesdata folder using Logstash 2.3.3 and sending them to ES 1.7.1. I was getting errors and fixing them, until logstash started without errors (as i think), but it… --- ## [Logstash file stops running at the point when it encounters stdout.read](https://discuss.elastic.co/t/logstash-file-stops-running-at-the-point-when-it-encounters-stdout-read/53660)
**Author:** [@KaranPanchal](https://discuss.elastic.co/u/KaranPanchal)\ **Replies:** 1\ **Last updated:** [July 4, 2016, 6:37pm UTC](https://discuss.elastic.co/t/logstash-file-stops-running-at-the-point-when-it-encounters-stdout-read/53660 "2016-07-04T18:37:16Z")
I am running a logstash file to extract new fields from my csv file. I have written a python script that will bring the required value from my csv file. This python script will be executed in the ruby plugin in the filte… --- ## [Cannot get custom fields to load in Filebeat](https://discuss.elastic.co/t/cannot-get-custom-fields-to-load-in-filebeat/53353)
**Author:** [@oxh10](https://discuss.elastic.co/u/oxh10)\ **Replies:** 4\ **Last updated:** [July 4, 2016, 6:32pm UTC](https://discuss.elastic.co/t/cannot-get-custom-fields-to-load-in-filebeat/53353 "2016-07-04T18:32:47Z")
Hello, I am having issues getting my logs parsed correctly in filebeat. In short, none of my custom fields (created by grok) are being loaded into the filebeat indicies page. The logs are being passed successfully but… --- ## [Logstash performance drop with high CPU usage](https://discuss.elastic.co/t/logstash-performance-drop-with-high-cpu-usage/53691)
**Author:** [@Nitesh\_Jain](https://discuss.elastic.co/u/Nitesh_Jain)\ **Replies:** 1\ **Last updated:** [July 4, 2016, 6:28pm UTC](https://discuss.elastic.co/t/logstash-performance-drop-with-high-cpu-usage/53691 "2016-07-04T18:28:55Z")
Hi, CPU usage spiked up to 3000% with the load of the box being around 400. There was no config change. This is also causing messages to pile up in the queues This is my setup: rsyslog -\> logstash -\> rabbitmq -\> logsta… --- ## [Mutate decimal to long with implied decimal places](https://discuss.elastic.co/t/mutate-decimal-to-long-with-implied-decimal-places/53694)
**Author:** [@mwilicap](https://discuss.elastic.co/u/mwilicap)\ **Replies:** 1\ **Last updated:** [July 4, 2016, 6:27pm UTC](https://discuss.elastic.co/t/mutate-decimal-to-long-with-implied-decimal-places/53694 "2016-07-04T18:27:08Z")
Hello. Is there a way in logstash to mutate an input field which is a string representation of a decimal (eg. "2387.74") to a long representation of the number with implied decimal places? Ie. We'd like to store that s… --- ## [Add Execption for special character from json log](https://discuss.elastic.co/t/add-execption-for-special-character-from-json-log/54285)
**Author:** [@infinitetutts](https://discuss.elastic.co/u/infinitetutts)\ **Replies:** 5\ **Last updated:** [July 4, 2016, 1:59pm UTC](https://discuss.elastic.co/t/add-execption-for-special-character-from-json-log/54285 "2016-07-04T13:59:37Z")
Hey Guys, I have been having a few frustrating days trying to crack this. I please need a solution either to fix the Ruby code or another method to add an exception for the escape character which is '\\\\x'. Logstash fi… --- ## [Logstash stalls with grok pattern with optional match](https://discuss.elastic.co/t/logstash-stalls-with-grok-pattern-with-optional-match/47215)
**Author:** [@kerk1v](https://discuss.elastic.co/u/kerk1v)\ **Replies:** 5\ **Last updated:** [July 4, 2016, 1:53pm UTC](https://discuss.elastic.co/t/logstash-stalls-with-grok-pattern-with-optional-match/47215 "2016-07-04T13:53:34Z")
Hi there, Got the following error when trying to match an optional part in the logs. Seems related to Ruby, here is the full log I get when i hit Ctrl+c when Logstash stalls and starts taking every bit of CPU on my 4-… --- ## [Output to elasticsearch gives errors but works with stdout](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530)
**Author:** [@infinitetutts](https://discuss.elastic.co/u/infinitetutts)\ **Replies:** 8\ **Last updated:** [July 4, 2016, 1:53pm UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530 "2016-07-04T13:53:27Z")
I think the error could be related to the ruby code, how do I fix this ? Config: input { beats { port =\> 5044 ssl =\> false } } filter { if \[type\] == "apache" { ruby { code =\> " if event\['message'\] event\[… --- ## [@timestamp conversion](https://discuss.elastic.co/t/timestamp-conversion/54640)
**Author:** [@Craig\_Botha](https://discuss.elastic.co/u/Craig_Botha)\ **Replies:** 2\ **Last updated:** [July 4, 2016, 11:58am UTC](https://discuss.elastic.co/t/timestamp-conversion/54640 "2016-07-04T11:58:57Z")
Hello Logstash community I set up ELK (filebeat\>logstash\>elasticsearch\>kibana) on a Windows 2012 server last week, and have been trying to configure Logstash to parse my logs correctly. I've got it mostly correct, but s… --- ## [Make future prediction on current cpu usage data with logstash](https://discuss.elastic.co/t/make-future-prediction-on-current-cpu-usage-data-with-logstash/52347)
**Author:** [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\ **Replies:** 8\ **Last updated:** [July 4, 2016, 10:59am UTC](https://discuss.elastic.co/t/make-future-prediction-on-current-cpu-usage-data-with-logstash/52347 "2016-07-04T10:59:57Z")
I have written a conf file to collect log file data about cpu usage. Is there anyway to do prediction of my cpu usage in the future 30 days with my current cpu usage data in the logstash conf file? --- ## [Logstash 5 is not outputting filebeat input and Kafka input](https://discuss.elastic.co/t/logstash-5-is-not-outputting-filebeat-input-and-kafka-input/54631)
**Author:** [@gaurav\_S\_javheri](https://discuss.elastic.co/u/gaurav_S_javheri)\ **Replies:** 0\ **Last updated:** [July 4, 2016, 10:14am UTC](https://discuss.elastic.co/t/logstash-5-is-not-outputting-filebeat-input-and-kafka-input/54631 "2016-07-04T10:14:36Z")
Below is configuration on Logstash 5 server input { kafka { topics =\> \["logstash\_logs"\] bootstrap\_servers =\> "zk\_server:2181" codec =\> plain { format =\> "%{message}" } … --- ## [LS can't connect to ES with Shield](https://discuss.elastic.co/t/ls-cant-connect-to-es-with-shield/34756)
**Author:** [@bluethundr](https://discuss.elastic.co/u/bluethundr)\ **Replies:** 12\ **Last updated:** [July 4, 2016, 8:34am UTC](https://discuss.elastic.co/t/ls-cant-connect-to-es-with-shield/34756 "2016-07-04T08:34:37Z")
Hey all, I've recently installed a new ES cluster with version 2.0, and Logstash at 1.5.5. This was done on 3 Centos 7.1 hosts. I'm running shield on this setup. Kibana is running behind nginx on the 1st node. I th… --- ## [How to use preference search on the elasticsearch input plugin?](https://discuss.elastic.co/t/how-to-use-preference-search-on-the-elasticsearch-input-plugin/54549)
**Author:** [@esamudio](https://discuss.elastic.co/u/esamudio)\ **Replies:** 1\ **Last updated:** [July 3, 2016, 12:52am UTC](https://discuss.elastic.co/t/how-to-use-preference-search-on-the-elasticsearch-input-plugin/54549 "2016-07-03T00:52:43Z")
I'm interested in the preference search to query by shard. Is there a way to do that in the elasticsearch input plugin? In case that's not possible, I could simply edit the plugin and reinstall it? I presume it shouldn… --- ## [Logstash JDBC Plugin Cassandra Driver Not Loaded Error](https://discuss.elastic.co/t/logstash-jdbc-plugin-cassandra-driver-not-loaded-error/38210)
**Author:** [@Vijay\_Dodla](https://discuss.elastic.co/u/Vijay_Dodla)\ **Replies:** 12\ **Last updated:** [July 2, 2016, 11:14am UTC](https://discuss.elastic.co/t/logstash-jdbc-plugin-cassandra-driver-not-loaded-error/38210 "2016-07-02T11:14:05Z")
Hi All Can anyone point me to examples on how logstash can read cassandra DB using JDBC . I tried something like this from logstash installed directory Step 1 : Create config file for logstash (File name :simple-out… --- ## [Logstash - Cassandra error-NoHostAvailableException](https://discuss.elastic.co/t/logstash-cassandra-error-nohostavailableexception/53923)
**Author:** [@Srivastavark](https://discuss.elastic.co/u/Srivastavark)\ **Replies:** 5\ **Last updated:** [July 1, 2016, 8:47pm UTC](https://discuss.elastic.co/t/logstash-cassandra-error-nohostavailableexception/53923 "2016-07-01T20:47:07Z")
Hello Experts, I am new to logstash and trying to index Cassandra data in Elasticsearch using JDBC input plugin but facing issue in test Logstash 2.3.3, Cassandra -3.5 Pipeline aborted due to error {:exception=\># … --- ## [Error: undefined method \`\[\]' for nil:NilClass in logstash](https://discuss.elastic.co/t/error-undefined-method-for-nil-nilclass-in-logstash/54234)
**Author:** [@hongyuan1306](https://discuss.elastic.co/u/hongyuan1306)\ **Replies:** 5\ **Last updated:** [July 1, 2016, 7:54pm UTC](https://discuss.elastic.co/t/error-undefined-method-for-nil-nilclass-in-logstash/54234 "2016-07-01T19:54:01Z")
With logstash 2.3.3 I encounter the following error with elasticssearch input plugin: Error: undefined method\[\]' for nil:NilClass\` Following is my logstash config file: input { elasticsearch { hosts =\> 'vhost… --- ## [Twitter input](https://discuss.elastic.co/t/twitter-input/54284)
**Author:** [@Diego\_Gomez](https://discuss.elastic.co/u/Diego_Gomez)\ **Replies:** 14\ **Last updated:** [July 1, 2016, 7:52pm UTC](https://discuss.elastic.co/t/twitter-input/54284 "2016-07-01T19:52:47Z")
this is my code logastash.conf. The problem I have is that there is no index data input { twitter { consumer\_key =\> "dplzZiWV2qP5q2dMZpM8lLb0c" consumer\_secret =\>"wpemcks8EqdgY3ZjV8dMJOViB… --- ## [What permissions does logstash need to run as a service?](https://discuss.elastic.co/t/what-permissions-does-logstash-need-to-run-as-a-service/52753)
**Author:** [@Edely\_Gomes](https://discuss.elastic.co/u/Edely_Gomes)\ **Replies:** 3\ **Last updated:** [July 1, 2016, 5:51pm UTC](https://discuss.elastic.co/t/what-permissions-does-logstash-need-to-run-as-a-service/52753 "2016-07-01T17:51:41Z")
I am running Logstash as a service, but it doesn't collect the logs. But when I run the bynary (bin/logstash -f test.conf) I collect the logs. And in the two cases I am using the same .conf file. In other posts it is… --- ## [Xml filter generates arrays with one value](https://discuss.elastic.co/t/xml-filter-generates-arrays-with-one-value/54314)
**Author:** [@sagizvi](https://discuss.elastic.co/u/sagizvi)\ **Replies:** 3\ **Last updated:** [July 1, 2016, 5:50pm UTC](https://discuss.elastic.co/t/xml-filter-generates-arrays-with-one-value/54314 "2016-07-01T17:50:09Z")
Hi I'm using an xml filter on data containing deep tags that sometimes contains arrays and sometimes dont. The xml filter generates the json with arrays, even when there is only one value. for example: "Header" =\> \[ … --- ## [Logstash configtest error](https://discuss.elastic.co/t/logstash-configtest-error/54547)
**Author:** [@dogelorde](https://discuss.elastic.co/u/dogelorde)\ **Replies:** 1\ **Last updated:** [July 1, 2016, 4:51pm UTC](https://discuss.elastic.co/t/logstash-configtest-error/54547 "2016-07-01T16:51:57Z")
Running an ELK Stack on Ubuntu Server 16 - Logstash 2.2.4 service is active, so it seems to load fine. When I run "sudo service logstash configtest", my output is this: Error: Expected one of #, input, filter, output at… --- ## [Reindexing depth-first, not breadth-first](https://discuss.elastic.co/t/reindexing-depth-first-not-breadth-first/54553)
**Author:** [@mhughes](https://discuss.elastic.co/u/mhughes)\ **Replies:** 0\ **Last updated:** [July 1, 2016, 4:35pm UTC](https://discuss.elastic.co/t/reindexing-depth-first-not-breadth-first/54553 "2016-07-01T16:35:35Z")
I'm using the elasticsearch input/output plugin to reindex my time-based indicies. Sometimes a reindex will fail half-way through. Unfortunately, there doesn't appear to be a good way to resume a reindex. If I've ind… --- ## [Logstash multiline filter pattern match -- needing help please \*\*Solved \*\*](https://discuss.elastic.co/t/logstash-multiline-filter-pattern-match-needing-help-please-solved/54440)
**Author:** [@davehatma](https://discuss.elastic.co/u/davehatma)\ **Replies:** 0\ **Last updated:** [June 30, 2016, 4:01pm UTC](https://discuss.elastic.co/t/logstash-multiline-filter-pattern-match-needing-help-please-solved/54440 "2016-06-30T16:01:35Z")
I was testing multiline in file section but DID NOT have a filter section. I included an empty filter section and its working. I am trying to get LogStash to combine a log file with multilines defining an event. The … --- ## [How to continuously pull from Cassandra into Logstash](https://discuss.elastic.co/t/how-to-continuously-pull-from-cassandra-into-logstash/44674)
**Author:** [@nkanov](https://discuss.elastic.co/u/nkanov)\ **Replies:** 3\ **Last updated:** [July 1, 2016, 1:33pm UTC](https://discuss.elastic.co/t/how-to-continuously-pull-from-cassandra-into-logstash/44674 "2016-07-01T13:33:34Z")
Hi all, I'm trying to setup Logstash to continuously pull information from Cassandra, in the same fashion it works with file inputs. I'm using the Cassandra JDBC Driver and I can successfully pull the contents, but onc… --- ## [Parse JSON array to flat JSON with filter, or alternative solution](https://discuss.elastic.co/t/parse-json-array-to-flat-json-with-filter-or-alternative-solution/54411)
**Author:** [@webmutation](https://discuss.elastic.co/u/webmutation)\ **Replies:** 3\ **Last updated:** [July 1, 2016, 12:53pm UTC](https://discuss.elastic.co/t/parse-json-array-to-flat-json-with-filter-or-alternative-solution/54411 "2016-07-01T12:53:29Z")
Hi, Im currently using log4j and kafka to send logs to ELK. I stumbled upon an issue that i am finding a bit strange there is no previous solution, so I am perhaps misusing the system. After reading a few dozen topics i… --- ## [Getting the following error while running lostash with docker](https://discuss.elastic.co/t/getting-the-following-error-while-running-lostash-with-docker/54495)
**Author:** [@vivc](https://discuss.elastic.co/u/vivc)\ **Replies:** 5\ **Last updated:** [July 1, 2016, 7:23am UTC](https://discuss.elastic.co/t/getting-the-following-error-while-running-lostash-with-docker/54495 "2016-07-01T07:23:45Z")
Errno::EACCES: Permission denied - /output mkdir at org/jruby/RubyDir.java:458 fu\_mkdir at /opt/logstash/vendor/jruby/lib/ruby/1.9/fileutils.rb:247 mkdir\_p at /opt/logstash/v… --- ## [How to prevent logstash service from auto-startingon centos6](https://discuss.elastic.co/t/how-to-prevent-logstash-service-from-auto-startingon-centos6/54124)
**Author:** [@sharath3185](https://discuss.elastic.co/u/sharath3185)\ **Replies:** 2\ **Last updated:** [July 1, 2016, 6:54am UTC](https://discuss.elastic.co/t/how-to-prevent-logstash-service-from-auto-startingon-centos6/54124 "2016-07-01T06:54:20Z")
I have some custom start/stop scripts to start the logstash and elasticsearch services on my centos system. I want elasticsearch service to be started and should be ready to accept the events from logstash before the lo… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1131) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1133)