# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=867 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 868 --- ## [\[GEOIP\] Get target value as geoip from json input](https://discuss.elastic.co/t/geoip-get-target-value-as-geoip-from-json-input/109198)
**Author:** [@cs0sf](https://discuss.elastic.co/u/cs0sf)\ **Replies:** 6\ **Last updated:** [November 28, 2017, 7:28am UTC](https://discuss.elastic.co/t/geoip-get-target-value-as-geoip-from-json-input/109198 "2017-11-28T07:28:50Z")
Hi, I am trying to make use of geoip plugin so that the IP field could be mapped to geo location, etc My sample log looks like the following: {"key1":"blah", "network":\[{"IP":"80.80.229.213"},{"IP":"80.80.229.216"}\]} … --- ## [How to config the shard numbers when index created by logstash?](https://discuss.elastic.co/t/how-to-config-the-shard-numbers-when-index-created-by-logstash/109003)
**Author:** [@Sungkeun\_Lee](https://discuss.elastic.co/u/Sungkeun_Lee)\ **Replies:** 2\ **Last updated:** [November 28, 2017, 7:14am UTC](https://discuss.elastic.co/t/how-to-config-the-shard-numbers-when-index-created-by-logstash/109003 "2017-11-28T07:14:31Z")
Hi, I am using ELK for collecting log. Logstash create a index everyday. Default shard setting is 5, but I wish change that. How do I have to do? Please let me know. --- ## [Grok two words as one term](https://discuss.elastic.co/t/grok-two-words-as-one-term/109305)
**Author:** [@Xuzheng\_Lin](https://discuss.elastic.co/u/Xuzheng_Lin)\ **Replies:** 2\ **Last updated:** [November 28, 2017, 6:58am UTC](https://discuss.elastic.co/t/grok-two-words-as-one-term/109305 "2017-11-28T06:58:59Z")
Hi, I am trying to grok below log with logstash and fetch the term "Received disconnect". (My ELK Version is still 5.1.1 at the moment) Nov 27 00:00:00 ubuntu sshd\[123456\]: Received disconnect from 111.111.111.111: 14:… --- ## [Logstash http output plugin 6.0 ssl issue](https://discuss.elastic.co/t/logstash-http-output-plugin-6-0-ssl-issue/109317)
**Author:** [@suresh\_subbaiah](https://discuss.elastic.co/u/suresh_subbaiah)\ **Replies:** 0\ **Last updated:** [November 28, 2017, 6:45am UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-6-0-ssl-issue/109317 "2017-11-28T06:45:11Z")
Hi, We are migrating from logstash 2.x to logstash 6.0 with http output plugin. Its a secure request we are making from logstash however we are getting the below error with the latest version. \[2017-11-28T06:35:59,558\]… --- ## [The date filter's timezone offset match not working?](https://discuss.elastic.co/t/the-date-filters-timezone-offset-match-not-working/109303)
**Author:** [@helloEson](https://discuss.elastic.co/u/helloEson)\ **Replies:** 4\ **Last updated:** [November 28, 2017, 6:37am UTC](https://discuss.elastic.co/t/the-date-filters-timezone-offset-match-not-working/109303 "2017-11-28T06:37:33Z")
Hello, I have a time field like this: "time\_local" =\> "23/Nov/2017:18:54:31 +0800" And i want to use date filter to extract it to field @timestamp, so i have the filter pattern like this: date { match =\> \[ "time\_lo… --- ## [MySQL Syncing with Logstash to Elasticsearch](https://discuss.elastic.co/t/mysql-syncing-with-logstash-to-elasticsearch/109269)
**Author:** [@Moomba](https://discuss.elastic.co/u/Moomba)\ **Replies:** 1\ **Last updated:** [November 28, 2017, 6:31am UTC](https://discuss.elastic.co/t/mysql-syncing-with-logstash-to-elasticsearch/109269 "2017-11-28T06:31:17Z")
Hello, For the last few hours i have been trying to implement synchronisation between MySQL database and Elasticsearch using Logstash and the JDBC plugin. Each row in each table has a updated\_at column (timestamp, refr… --- ## [Help in Logstash](https://discuss.elastic.co/t/help-in-logstash/109258)
**Author:** [@Sridhar](https://discuss.elastic.co/u/Sridhar)\ **Replies:** 3\ **Last updated:** [November 28, 2017, 6:26am UTC](https://discuss.elastic.co/t/help-in-logstash/109258 "2017-11-28T06:26:06Z")
Hi All, I'm going to setup an elk stack to monitor logs. We have 4 websphere servers and 4 IBM http servers. Websphere Logs: SystemOut.log SystemErr.log Http Server logs: Access.log Error.log All these logs wil… --- ## [Logstash connections are going to close\_wait intermittently](https://discuss.elastic.co/t/logstash-connections-are-going-to-close-wait-intermittently/109311)
**Author:** [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\ **Replies:** 0\ **Last updated:** [November 28, 2017, 6:00am UTC](https://discuss.elastic.co/t/logstash-connections-are-going-to-close-wait-intermittently/109311 "2017-11-28T06:00:04Z")
Hi I'm using filebeat 5.1.2 version to send logs from around 430 instances to a central logstash-2.4.1 server over ssl and 443 port. I am seeing a scenario where sometimes when logstash is restarted, filebeat connections… --- ## [Logstash stop running](https://discuss.elastic.co/t/logstash-stop-running/109302)
**Author:** [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Replies:** 0\ **Last updated:** [November 28, 2017, 2:56am UTC](https://discuss.elastic.co/t/logstash-stop-running/109302 "2017-11-28T02:56:12Z")
this is my logstash config: input{ tcp{ port =\> 5510 codec =\> json type =\>"ntopng-\*" } } filter{ if\[type\]=="ntopng-\*" { if "" not in \[IPV4\_SRC\_ADDR\] and "" not in \[IPV6\_SRC\_ADDR\] { drop{} } } } output{ el… --- ## [Logstash 6.0 breaks winlogbeat agent indexing](https://discuss.elastic.co/t/logstash-6-0-breaks-winlogbeat-agent-indexing/108595)
**Author:** [@kwslavens](https://discuss.elastic.co/u/kwslavens)\ **Replies:** 2\ **Last updated:** [November 27, 2017, 11:57pm UTC](https://discuss.elastic.co/t/logstash-6-0-breaks-winlogbeat-agent-indexing/108595 "2017-11-27T23:57:12Z")
I've got a winlogbeat agents version 5.x sending data to a kafka queue with a topic name of winlogbeat. Our indexer cluster is pulling out of that topic. With 6.0 Elasticsearch and 5.6.4 agents / logstash 5.6.4 all is g… --- ## [Greater than not working in logstash filter and crashing](https://discuss.elastic.co/t/greater-than-not-working-in-logstash-filter-and-crashing/109249)
**Author:** [@zzurashvili](https://discuss.elastic.co/u/zzurashvili)\ **Replies:** 2\ **Last updated:** [November 27, 2017, 9:55pm UTC](https://discuss.elastic.co/t/greater-than-not-working-in-logstash-filter-and-crashing/109249 "2017-11-27T21:55:24Z")
I'm getting IPFIX flow data and in filter I need to add fields and tags based on source or destination port. Everything is working perfectly when I compare port using "==". if I use "\>" I get: \[2017-11-27T11:18:45,502\]\[… --- ## [Log full of "encountered retryable error" and logstash stopped listening](https://discuss.elastic.co/t/log-full-of-encountered-retryable-error-and-logstash-stopped-listening/108193)
**Author:** [@Jose\_Ramierez](https://discuss.elastic.co/u/Jose_Ramierez)\ **Replies:** 4\ **Last updated:** [November 27, 2017, 9:32pm UTC](https://discuss.elastic.co/t/log-full-of-encountered-retryable-error-and-logstash-stopped-listening/108193 "2017-11-27T21:32:45Z")
Hmmm. We were running logstash 5.6.1, and we noticed it stopped listening on IP. So we upgraded it to logstash 5.6.4 And here I am , a few days later, and logstash stopped listening, again, on IP. Clients cannot conne… --- ## [Ingest problem with filebeat and logstash](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188)
**Author:** [@Rodolphe\_Redoute](https://discuss.elastic.co/u/Rodolphe_Redoute)\ **Replies:** 9\ **Last updated:** [November 27, 2017, 8:54pm UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188 "2017-11-27T20:54:33Z")
Hi, I have an ingest problem with filebeat and logstash. I have machines with a configuration on it's own to get different type of log files. my first filebeat.yml file is like this : filebeat.prospectors: - document\_… --- ## [Logstash filter debugging workflow](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632)
**Author:** [@ajhstn](https://discuss.elastic.co/u/ajhstn)\ **Replies:** 10\ **Last updated:** [November 27, 2017, 8:02pm UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632 "2017-11-27T20:02:47Z")
hello, how does one test {filters} quickly making small iterative changes to the .conf file and seeing results? Eg my current workflow is open the .conf file, make a change, save it, restart logstash, wait for a matchin… --- ## [Found duplicate records in elasticsearch](https://discuss.elastic.co/t/found-duplicate-records-in-elasticsearch/109071)
**Author:** [@nsphaniraj](https://discuss.elastic.co/u/nsphaniraj)\ **Replies:** 7\ **Last updated:** [November 27, 2017, 5:46pm UTC](https://discuss.elastic.co/t/found-duplicate-records-in-elasticsearch/109071 "2017-11-27T17:46:53Z")
Hello, We have setup logs pipeline as below filebeat -\> kafka -\> logstash -\> elasticsearch filebeat is running on many servers which will send to centralized kafka server one topic and 20 partitions. We have 2 logsta… --- ## [Output CEF](https://discuss.elastic.co/t/output-cef/109260)
**Author:** [@saad](https://discuss.elastic.co/u/saad)\ **Replies:** 0\ **Last updated:** [November 27, 2017, 5:30pm UTC](https://discuss.elastic.co/t/output-cef/109260 "2017-11-27T17:30:59Z")
Hi I created conf file in logstash to receive cef input and then send this event by using output cef. I got the cef event successfully as input. however, I found that the output is sent to acrsight connector but only w… --- ## [Redis -\> LS -\> ES WRONGTYPE Operation against a key holding the wrong kind of value](https://discuss.elastic.co/t/redis-ls-es-wrongtype-operation-against-a-key-holding-the-wrong-kind-of-value/109255)
**Author:** [@dseixas](https://discuss.elastic.co/u/dseixas)\ **Replies:** 0\ **Last updated:** [November 27, 2017, 5:15pm UTC](https://discuss.elastic.co/t/redis-ls-es-wrongtype-operation-against-a-key-holding-the-wrong-kind-of-value/109255 "2017-11-27T17:15:43Z")
Hi, We're trying to use the combo Redis+LS+ES to try to persist xAPI statements (these statements are JSON strings). So far, we write correctly in redis, but when LS try to do its thing we get this message: WRONGTYPE … --- ## [Read unstructured log file to extract certain values](https://discuss.elastic.co/t/read-unstructured-log-file-to-extract-certain-values/109253)
**Author:** [@telebh](https://discuss.elastic.co/u/telebh)\ **Replies:** 0\ **Last updated:** [November 27, 2017, 5:03pm UTC](https://discuss.elastic.co/t/read-unstructured-log-file-to-extract-certain-values/109253 "2017-11-27T17:03:59Z")
Hi there, I am newbie to logstash... trying to use to parse our dataloding log files, extraction certain values from the file and store it into a JSON formated out file. for example: below is a snippet of the text log f… --- ## [Logstash 6.0 breaks ruby code if running more than 1 worker](https://discuss.elastic.co/t/logstash-6-0-breaks-ruby-code-if-running-more-than-1-worker/109083)
**Author:** [@vb4t](https://discuss.elastic.co/u/vb4t)\ **Replies:** 6\ **Last updated:** [November 27, 2017, 4:13pm UTC](https://discuss.elastic.co/t/logstash-6-0-breaks-ruby-code-if-running-more-than-1-worker/109083 "2017-11-27T16:13:33Z")
I upgraded Logstash to 6.0. After the upgrade a lot of ruby code stopped to work. Following code worked in Logstash 5, but in Logstash 6 it ofter causes these errors: Ruby exception occurred: org.jruby.RubyNil cannot b… --- ## [How to call Ruby function in logstash filter?](https://discuss.elastic.co/t/how-to-call-ruby-function-in-logstash-filter/107930)
**Author:** [@Harim\_Tong](https://discuss.elastic.co/u/Harim_Tong)\ **Replies:** 2\ **Last updated:** [November 27, 2017, 3:55pm UTC](https://discuss.elastic.co/t/how-to-call-ruby-function-in-logstash-filter/107930 "2017-11-27T15:55:42Z")
I don't want to write too much Ruby code in filter(Ruby filter) ,because it's hard to maintenance, can logstash filter call user defined function? Like I have a business function: foo(x) in the file: my.rb as below: d… --- ## [Logstash rss plugin not storing field](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447)
**Author:** [@prophoto](https://discuss.elastic.co/u/prophoto)\ **Replies:** 14\ **Last updated:** [November 27, 2017, 2:02pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447 "2017-11-27T14:02:23Z")
I have successfully setup the logstash-rss-plugin and its working fine except one field is not being stored. I need it since the feed items are updated regularly, and without it I am getting duplicate entries in Elastics… --- ## [What is the difference between stdin and file in logstash](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910)
**Author:** [@yishain11](https://discuss.elastic.co/u/yishain11)\ **Replies:** 8\ **Last updated:** [November 27, 2017, 12:52pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910 "2017-11-27T12:52:34Z")
Hi I have two configurations, one with the input as stdin, and one using file path. both inputs are JSON structured text files. The filters are the same (using JSON encode, json and split), the output is the same. Wh… --- ## [Logstash input dynamodb plugin](https://discuss.elastic.co/t/logstash-input-dynamodb-plugin/109194)
**Author:** [@dangal](https://discuss.elastic.co/u/dangal)\ **Replies:** 4\ **Last updated:** [November 27, 2017, 11:59am UTC](https://discuss.elastic.co/t/logstash-input-dynamodb-plugin/109194 "2017-11-27T11:59:54Z")
Hi all, why this logstash input dynamodb plugin is not published for 5.x and 6.x i am able to install that plugin successfully in 2.3 and 2.4 why can't i in 5.x and 6.x ? please respond to my question Thank you --- ## [Logstash: parsing snmp traps ( continue...)](https://discuss.elastic.co/t/logstash-parsing-snmp-traps-continue/109204)
**Author:** [@Robert.I](https://discuss.elastic.co/u/Robert.I)\ **Replies:** 1\ **Last updated:** [November 27, 2017, 11:33am UTC](https://discuss.elastic.co/t/logstash-parsing-snmp-traps-continue/109204 "2017-11-27T11:33:17Z")
Hello all, I would like to share my solution with my problem parsing traps. I my problem is described in this topic: https://discuss.elastic.co/t/logstash-parsing-snmp-traps/103208. The solution was : List item Cha… --- ## [Kafka output plugin not sending messages](https://discuss.elastic.co/t/kafka-output-plugin-not-sending-messages/108768)
**Author:** [@Senthilr2k](https://discuss.elastic.co/u/Senthilr2k)\ **Replies:** 2\ **Last updated:** [November 27, 2017, 10:48am UTC](https://discuss.elastic.co/t/kafka-output-plugin-not-sending-messages/108768 "2017-11-27T10:48:44Z")
Logstash kafka output plugin says the output is flushed. But the message is not available in kafka topic. I am able to submit message to kafka using the same configuration built using java. Can someone help please. --- ## ["Counters can not be incremented by negative values" error when using metric filters freezes workers and "corrupts" the persistent queue](https://discuss.elastic.co/t/counters-can-not-be-incremented-by-negative-values-error-when-using-metric-filters-freezes-workers-and-corrupts-the-persistent-queue/108644)
**Author:** [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Replies:** 1\ **Last updated:** [November 23, 2017, 5:41am UTC](https://discuss.elastic.co/t/counters-can-not-be-incremented-by-negative-values-error-when-using-metric-filters-freezes-workers-and-corrupts-the-persistent-queue/108644 "2017-11-23T05:41:26Z")
Hi, I am running Logstash-6.0.0 with metrics filters to track performance. I have noticed that performance of some of my nodes decreases over time. On checking the logs I found log messages like the following \[2017-1… --- ## [Healthcheck in logstash-http-plugin without authentication](https://discuss.elastic.co/t/healthcheck-in-logstash-http-plugin-without-authentication/108071)
**Author:** [@ziaur25](https://discuss.elastic.co/u/ziaur25)\ **Replies:** 1\ **Last updated:** [November 27, 2017, 9:24am UTC](https://discuss.elastic.co/t/healthcheck-in-logstash-http-plugin-without-authentication/108071 "2017-11-27T09:24:31Z")
I have configured my logstash with below configuration. With this, I am able to stream logs from cloudfoundry env. But in my case, I want to use a vendor loadbalancer in place for my logstash running in different machine… --- ## [What's the best filter for path style url?](https://discuss.elastic.co/t/whats-the-best-filter-for-path-style-url/109179)
**Author:** [@helloEson](https://discuss.elastic.co/u/helloEson)\ **Replies:** 2\ **Last updated:** [November 27, 2017, 9:03am UTC](https://discuss.elastic.co/t/whats-the-best-filter-for-path-style-url/109179 "2017-11-27T09:03:15Z")
I want to extract the params in a path style url, it's something like this: /cid/27/end/2017-12-15/format/json/pid/1600 and the result i want is like this: cid=27,end=2017-12-15,format=json,pid=1600 Is there any good … --- ## [Logstash 6.0 breaks ruby event iteration](https://discuss.elastic.co/t/logstash-6-0-breaks-ruby-event-iteration/109084)
**Author:** [@vb4t](https://discuss.elastic.co/u/vb4t)\ **Replies:** 4\ **Last updated:** [November 27, 2017, 6:52am UTC](https://discuss.elastic.co/t/logstash-6-0-breaks-ruby-event-iteration/109084 "2017-11-27T06:52:07Z")
After upgrade to LS 6.0 following code stopped to work properly: ruby { code =\> " hash = event.to\_hash hash.each do |k,v| if v == '-' event.remove(k) else if v.is\_a?… --- ## [Logstash-output-kinesis](https://discuss.elastic.co/t/logstash-output-kinesis/107655)
**Author:** [@abdalians](https://discuss.elastic.co/u/abdalians)\ **Replies:** 2\ **Last updated:** [November 27, 2017, 6:34am UTC](https://discuss.elastic.co/t/logstash-output-kinesis/107655 "2017-11-27T06:34:23Z")
Continuing the discussion from Missing logstash-output-kinesis for Logstash 5.2?: Repeating the question from @NirReuveny, why is there no logstash-output-kinesis plugin? [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=866) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=868)