# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1119 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1120 --- ## [GeoLite2 not working anymore on 5.0.0 alpha4](https://discuss.elastic.co/t/geolite2-not-working-anymore-on-5-0-0-alpha4/57267)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 0\ **Last updated:** [August 4, 2016, 5:39pm UTC](https://discuss.elastic.co/t/geolite2-not-working-anymore-on-5-0-0-alpha4/57267 "2016-08-04T17:39:04Z")
Hello, I have moved my elasticsearch cluster from docker based to host based for more simplicity and after I moved all the configuration files I could see that the GeoLite2 Databases are not working anymore. I had to s… --- ## [How to Test Beats +Logstash working fine or not?](https://discuss.elastic.co/t/how-to-test-beats-logstash-working-fine-or-not/57264)
**Author:** [@Gunjan\_Gupta](https://discuss.elastic.co/u/Gunjan_Gupta)\ **Replies:** 0\ **Last updated:** [August 4, 2016, 5:30pm UTC](https://discuss.elastic.co/t/how-to-test-beats-logstash-working-fine-or-not/57264 "2016-08-04T17:30:39Z")
Scenario: Server1: FileBeat+Logstash Currently,I have configured Filebeat and ran its service.After that I have configured logstash and ran it which is also working fine.Now both are in running status.r In logstas… --- ## [Dynamic topics for kafka output plugin](https://discuss.elastic.co/t/dynamic-topics-for-kafka-output-plugin/56429)
**Author:** [@ronnysh](https://discuss.elastic.co/u/ronnysh)\ **Replies:** 4\ **Last updated:** [August 4, 2016, 5:25pm UTC](https://discuss.elastic.co/t/dynamic-topics-for-kafka-output-plugin/56429 "2016-08-04T17:25:54Z")
Hi guys, assuming that i have the following json input: { "user": "1", "compay\_id": "100" } can i set compay\_id to kafka output topic\_id? thanks in advance, ronnysh --- ## [Xml filter plugin - creating nested field out of null object: Can't get text on a END\_OBJECT](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928)
**Author:** [@whyapenny](https://discuss.elastic.co/u/whyapenny)\ **Replies:** 10\ **Last updated:** [August 4, 2016, 4:08pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928 "2016-08-04T16:08:03Z")
FYI, Here is the error seen: "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse \[entry.AppId.raw\]", "caused\_by"=\>{"type"=\>"illegal\_state\_exception", "reason"=\>"Can't get text on a… --- ## [Logstash crashed with IdentityMapUpperLimitException](https://discuss.elastic.co/t/logstash-crashed-with-identitymapupperlimitexception/52394)
**Author:** [@baneshbanna](https://discuss.elastic.co/u/baneshbanna)\ **Replies:** 1\ **Last updated:** [August 4, 2016, 2:42pm UTC](https://discuss.elastic.co/t/logstash-crashed-with-identitymapupperlimitexception/52394 "2016-08-04T14:42:52Z")
Logstash crashed with the following error. LogStash::Codecs::IdentityMapCodec::IdentityMapUpperLimitException: LogStash::Codecs::IdentityMapCodec::IdentityMapUpperLimitException visit at /opt/logstash/ve… --- ## [Input plugin runtime dependency](https://discuss.elastic.co/t/input-plugin-runtime-dependency/57220)
**Author:** [@cmeerbeek](https://discuss.elastic.co/u/cmeerbeek)\ **Replies:** 0\ **Last updated:** [August 4, 2016, 1:14pm UTC](https://discuss.elastic.co/t/input-plugin-runtime-dependency/57220 "2016-08-04T13:14:15Z")
Another thing I cannot figure out. I have developed a logstash input plug-in but during installation I now have a depedency to logstash-core 5.... Why? My gemspec file contains: s.add\_runtime\_dependency "logstash-c… --- ## [Troubles getting custom input plugin with external gem working](https://discuss.elastic.co/t/troubles-getting-custom-input-plugin-with-external-gem-working/56171)
**Author:** [@cmeerbeek](https://discuss.elastic.co/u/cmeerbeek)\ **Replies:** 1\ **Last updated:** [August 4, 2016, 12:37pm UTC](https://discuss.elastic.co/t/troubles-getting-custom-input-plugin-with-external-gem-working/56171 "2016-08-04T12:37:03Z")
I am building a custom plug-in with a connection to the Google API. To do this I have included the google-api-client in my gemspec file; s.add\_runtime\_dependency 'google-api-client'. When I want to build and test my plug… --- ## [Logstash does not show any output on windows with stdout](https://discuss.elastic.co/t/logstash-does-not-show-any-output-on-windows-with-stdout/56491)
**Author:** [@a.madhusudan](https://discuss.elastic.co/u/a.madhusudan)\ **Replies:** 9\ **Last updated:** [August 4, 2016, 11:37am UTC](https://discuss.elastic.co/t/logstash-does-not-show-any-output-on-windows-with-stdout/56491 "2016-08-04T11:37:46Z")
I am using logstash 2.3.4. I tried to parse a file and output it to stdout. Nothing fancy here. Given below is the config file \` input { file { path =\> "/c/logstash-2.3.4/input.txt" start\_positio… --- ## [Logstash crashes after failed dns lookup](https://discuss.elastic.co/t/logstash-crashes-after-failed-dns-lookup/55223)
**Author:** [@renevdm](https://discuss.elastic.co/u/renevdm)\ **Replies:** 4\ **Last updated:** [August 4, 2016, 11:24am UTC](https://discuss.elastic.co/t/logstash-crashes-after-failed-dns-lookup/55223 "2016-08-04T11:24:13Z")
Every now and then my logstash crashes randomly. having a look at the logs, it seems to be related to a failed DNS lookup. {:timestamp=\>"2016-07-11T20:30:04.574000+0200", :message=\>"Exception in pipelineworker, the pip… --- ## [Logstash kv and regex](https://discuss.elastic.co/t/logstash-kv-and-regex/57117)
**Author:** [@dihpierrick](https://discuss.elastic.co/u/dihpierrick)\ **Replies:** 4\ **Last updated:** [August 4, 2016, 9:06am UTC](https://discuss.elastic.co/t/logstash-kv-and-regex/57117 "2016-08-04T09:06:32Z")
Hi, I have got a question concerning the KV filter. Is it possible to use an regexp to get only the key matching the regex ? ? For example, if the message is : t\_1=qsdfgh t\_2=ploki p\_1=nbvcx t\_3=azerty I would l… --- ## [2 codec in one filter plugin](https://discuss.elastic.co/t/2-codec-in-one-filter-plugin/57093)
**Author:** [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\ **Replies:** 2\ **Last updated:** [August 4, 2016, 8:58am UTC](https://discuss.elastic.co/t/2-codec-in-one-filter-plugin/57093 "2016-08-04T08:58:13Z")
Hi! I would like to ask question about filter plugin I am using logstash 5.0.0 alpha4 I am in a situation where I have to set multiline codec and plain codec in single file module. This seems to not work since logs… --- ## [Log Correlation](https://discuss.elastic.co/t/log-correlation/57187)
**Author:** [@NicoYUE](https://discuss.elastic.co/u/NicoYUE)\ **Replies:** 1\ **Last updated:** [August 4, 2016, 8:13am UTC](https://discuss.elastic.co/t/log-correlation/57187 "2016-08-04T08:13:31Z")
Hello, I'm currently trying to use Logstash filter to correlate logs before outputting the result into Elasticsearch. I thought about using aggregate when the logs are in a good order like a b c into "abc". But… --- ## [Logstash Plugins Community Maintainers](https://discuss.elastic.co/t/logstash-plugins-community-maintainers/35953)
**Author:** [@alvin](https://discuss.elastic.co/u/alvin)\ **Replies:** 8\ **Last updated:** [August 4, 2016, 8:08am UTC](https://discuss.elastic.co/t/logstash-plugins-community-maintainers/35953 "2016-08-04T08:08:39Z")
Greetings Logstash Community! Hearty thanks to everyone in the community who has contributed and helped make the Logstash community so healthy and successful. Whether it be issues, PRs, or any other suggestions around … --- ## [Why codec dots not working](https://discuss.elastic.co/t/why-codec-dots-not-working/57133)
**Author:** [@achenchik\_nexters](https://discuss.elastic.co/u/achenchik_nexters)\ **Replies:** 1\ **Last updated:** [August 4, 2016, 8:02am UTC](https://discuss.elastic.co/t/why-codec-dots-not-working/57133 "2016-08-04T08:02:43Z")
Hello there, Any idea why codec docs is not working at all? \`input { file { path =\> "/opt/logstash/prod-debug/\*.log" start\_position =\> "beginning" ignore\_older =\> 2678400 } tcp { port =\> 5000 } } filter… --- ## [Fields added using a ruby filter doesn't survive split filter](https://discuss.elastic.co/t/fields-added-using-a-ruby-filter-doesnt-survive-split-filter/57158)
**Author:** [@kaung](https://discuss.elastic.co/u/kaung)\ **Replies:** 1\ **Last updated:** [August 4, 2016, 4:30am UTC](https://discuss.elastic.co/t/fields-added-using-a-ruby-filter-doesnt-survive-split-filter/57158 "2016-08-04T04:30:18Z")
I have a logstash config looking roughly like below: #ruby \<1\> ruby { code =\> "event\['@metadata'\]\['scheduledTime'\] = Time.now" } ... split { field =\> "xxx" target =\> "yyy" } ... #ruby \<2\> … --- ## [How to configure logstash to read only new entries from database but not using "sql\_last\_start"](https://discuss.elastic.co/t/how-to-configure-logstash-to-read-only-new-entries-from-database-but-not-using-sql-last-start/29140)
**Author:** [@IgorG](https://discuss.elastic.co/u/IgorG)\ **Replies:** 3\ **Last updated:** [August 3, 2016, 8:23pm UTC](https://discuss.elastic.co/t/how-to-configure-logstash-to-read-only-new-entries-from-database-but-not-using-sql-last-start/29140 "2016-08-03T20:23:55Z")
Is it possible to query database like: "SELECT \* FROM TABLE\_NAME where id \> : last\_saved\_id" and not by using built-in parameter "sql\_last\_start" e.g. "SELECT \* FROM TABLE\_NAME where timestamp \> : sql\_last\_start". I don'… --- ## [Increased Kafka Lag due to load balancing](https://discuss.elastic.co/t/increased-kafka-lag-due-to-load-balancing/56041)
**Author:** [@stecino](https://discuss.elastic.co/u/stecino)\ **Replies:** 1\ **Last updated:** [August 3, 2016, 6:11pm UTC](https://discuss.elastic.co/t/increased-kafka-lag-due-to-load-balancing/56041 "2016-08-03T18:11:54Z")
Hello, I am running logstash 2.2.2 and I recently upgraded kafka input/output 3.x plugin to support consumer offset storage to Kafka 0.9 As a result, I am connecting to brokers as oppose to Zookeepers. But what I have … --- ## [Decorate\_events in Logstash 5 kafka input](https://discuss.elastic.co/t/decorate-events-in-logstash-5-kafka-input/57128)
**Author:** [@Tsury](https://discuss.elastic.co/u/Tsury)\ **Replies:** 1\ **Last updated:** [August 3, 2016, 6:08pm UTC](https://discuss.elastic.co/t/decorate-events-in-logstash-5-kafka-input/57128 "2016-08-03T18:08:26Z")
Was this setting removed? I can't seem to find it in the documentation and it causes a config error when used. I have a Kafka -\> Logstash -\>Elasticsearch setup and I would like to set the ES index according to the Kafka … --- ## [Getting Elasticsearch index from Logstash message content](https://discuss.elastic.co/t/getting-elasticsearch-index-from-logstash-message-content/57129)
**Author:** [@Tsury](https://discuss.elastic.co/u/Tsury)\ **Replies:** 1\ **Last updated:** [August 3, 2016, 6:06pm UTC](https://discuss.elastic.co/t/getting-elasticsearch-index-from-logstash-message-content/57129 "2016-08-03T18:06:59Z")
Hi. I have a FileBeat -\> Kafka -\> Logstash -\> Elasticsearch setup, and the file that FileBeat monitors produces JSON lines with { "index" : "myIndex" }. Everything is sent to Kafka to the same topic and from there to L… --- ## [Determining Elasticsearch index from message](https://discuss.elastic.co/t/determining-elasticsearch-index-from-message/57121)
**Author:** [@Tsury](https://discuss.elastic.co/u/Tsury)\ **Replies:** 1\ **Last updated:** [August 3, 2016, 6:05pm UTC](https://discuss.elastic.co/t/determining-elasticsearch-index-from-message/57121 "2016-08-03T18:05:40Z")
Hi. I am getting the following message from a Kafka input and would like to determine the Elasticsearch index from it: { "@timestamp": "2016-08-03T14:58:39.964Z", "@version": "1", "message": "{\\"@timestamp\\":\\"… --- ## [What happen when logstash kafka input die?](https://discuss.elastic.co/t/what-happen-when-logstash-kafka-input-die/56056)
**Author:** [@xluffy](https://discuss.elastic.co/u/xluffy)\ **Replies:** 1\ **Last updated:** [August 3, 2016, 5:59pm UTC](https://discuss.elastic.co/t/what-happen-when-logstash-kafka-input-die/56056 "2016-08-03T17:59:23Z")
I use logstash for collecting log from Kafka and output to file. My logstash like a consumer. In document of Kafka, i see auto.commit.enable for detect position log when consumer fail. With logstash-input-kafka plugin, i… --- ## [Field created as float still shows as string in kibana Settings](https://discuss.elastic.co/t/field-created-as-float-still-shows-as-string-in-kibana-settings/57060)
**Author:** [@zoplex](https://discuss.elastic.co/u/zoplex)\ **Replies:** 3\ **Last updated:** [August 3, 2016, 3:59pm UTC](https://discuss.elastic.co/t/field-created-as-float-still-shows-as-string-in-kibana-settings/57060 "2016-08-03T15:59:28Z")
HI, I am creating field as float type on the fly in logstash: mutate { add\_field =\> { "slowq\_time" =\> {} } convert =\> { "slowq\_time" =\> "float" } } grok { patterns\_dir =\> \["/elk/lo… --- ## [\[Resolved\] Ruby filter exceptions in 5.0.0](https://discuss.elastic.co/t/resolved-ruby-filter-exceptions-in-5-0-0/56643)
**Author:** [@Rom1](https://discuss.elastic.co/u/Rom1)\ **Replies:** 1\ **Last updated:** [August 3, 2016, 3:16pm UTC](https://discuss.elastic.co/t/resolved-ruby-filter-exceptions-in-5-0-0/56643 "2016-08-03T15:16:36Z")
Hi, I want to create a new field by computing hour, minute and second in seconds with logstash 5.0.0-alpha4. I'm using : ruby { code =\> "event\['duration'\] = event\['dur\_hour'\] \* 3600 + event\['dur\_min'\] \* 60 + event\['… --- ## [Having problem with Logstash filter aggregate example](https://discuss.elastic.co/t/having-problem-with-logstash-filter-aggregate-example/57092)
**Author:** [@vishwanathh](https://discuss.elastic.co/u/vishwanathh)\ **Replies:** 1\ **Last updated:** [August 3, 2016, 1:51pm UTC](https://discuss.elastic.co/t/having-problem-with-logstash-filter-aggregate-example/57092 "2016-08-03T13:51:20Z")
Hello, I'm not getting the desired output for this example that I"m testing on my windows machine. file: sqllog INFO - 12345 - TASK\_START - start INFO - 12345 - SQL - sqlQuery1 - 12 INFO - 12345 - SQL - sqlQuery2 - 3… --- ## [\[SOLVED\]Logstash strange indexing error](https://discuss.elastic.co/t/solved-logstash-strange-indexing-error/46702)
**Author:** [@mikula](https://discuss.elastic.co/u/mikula)\ **Replies:** 5\ **Last updated:** [August 3, 2016, 12:51pm UTC](https://discuss.elastic.co/t/solved-logstash-strange-indexing-error/46702 "2016-08-03T12:51:30Z")
Hello, During reindexing through logstash I get this error (lot of times...): Should not be that message part also qouted? Failed action. {:status=\>400, :action=\>\["index", {:\_id=\>"+lmAQBUf3Mrk9Yymy08bABnbE8mg8EFRHqrx6… --- ## [Suricata timestamp problem](https://discuss.elastic.co/t/suricata-timestamp-problem/57037)
**Author:** [@Slydder](https://discuss.elastic.co/u/Slydder)\ **Replies:** 3\ **Last updated:** [August 3, 2016, 9:44am UTC](https://discuss.elastic.co/t/suricata-timestamp-problem/57037 "2016-08-03T09:44:22Z")
Hey all, I am sendling suricata json to logstash and logstash is having problems with the timestamp format: {:timestamp=\>"2016-08-02T18:28:37.516000+0200", :message=\>"Failed parsing date from field", :field=\>"timesta… --- ## [Substring in logstash](https://discuss.elastic.co/t/substring-in-logstash/57070)
**Author:** [@vandana\_sethi](https://discuss.elastic.co/u/vandana_sethi)\ **Replies:** 0\ **Last updated:** [August 3, 2016, 5:27am UTC](https://discuss.elastic.co/t/substring-in-logstash/57070 "2016-08-03T05:27:07Z")
I have a field enddate = "20160801105021" I want to create a new field which must contain 8 character of the enddate or must have 0-7 character like newdate = "20160801" --- ## [Host down in hosts list](https://discuss.elastic.co/t/host-down-in-hosts-list/56933)
**Author:** [@stecino](https://discuss.elastic.co/u/stecino)\ **Replies:** 5\ **Last updated:** [August 3, 2016, 2:04am UTC](https://discuss.elastic.co/t/host-down-in-hosts-list/56933 "2016-08-03T02:04:01Z")
Hello, What happens if a elastic search data node defined in the hosts list for elastic search output definition is down? Will I get an error can't connect to ES, or will it try to find available data node to make the … --- ## [Logstash input from tcp/udp protocol](https://discuss.elastic.co/t/logstash-input-from-tcp-udp-protocol/57055)
**Author:** [@bishaka](https://discuss.elastic.co/u/bishaka)\ **Replies:** 7\ **Last updated:** [August 2, 2016, 11:48pm UTC](https://discuss.elastic.co/t/logstash-input-from-tcp-udp-protocol/57055 "2016-08-02T23:48:44Z")
Hi, I am trying to send some JSON formatted logs to elasticsearch over tcp/udp protocol for visualizations in Kibana. I have the config file below: input { tcp { host =\> "http://nimhts23.hm230.org" port =\> 89… --- ## [Do plugins get upgraded along side Logstash?](https://discuss.elastic.co/t/do-plugins-get-upgraded-along-side-logstash/52924)
**Author:** [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Replies:** 2\ **Last updated:** [August 2, 2016, 11:15pm UTC](https://discuss.elastic.co/t/do-plugins-get-upgraded-along-side-logstash/52924 "2016-08-02T23:15:14Z")
A couple weeks ago my indexer was going nuts. Eventually I fixed it by manually upgrading the plugins. Which confuses me because I would think Logstash core plugins would get upgraded at the same time as the main progra… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1118) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1120)