# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=606 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 607 --- ## [Parsing incoming Json Data](https://discuss.elastic.co/t/parsing-incoming-json-data/166952)
**Author:** [@aseemmittal](https://discuss.elastic.co/u/aseemmittal)\ **Replies:** 3\ **Last updated:** [February 5, 2019, 10:38am UTC](https://discuss.elastic.co/t/parsing-incoming-json-data/166952 "2019-02-05T10:38:15Z")
Hi, I have following incoming data {"event":\[{"duration":"3","time":"2019-02-04 17:49:27","event":"achievementUnlock","message":"Achievement unlocked"},{"duration":"5","time":"2019-02-04 17:49:27","event":"achievementU… --- ## [Grokparsefailure randomly](https://discuss.elastic.co/t/grokparsefailure-randomly/166203)
**Author:** [@fry2k](https://discuss.elastic.co/u/fry2k)\ **Replies:** 9\ **Last updated:** [February 5, 2019, 10:12am UTC](https://discuss.elastic.co/t/grokparsefailure-randomly/166203 "2019-02-05T10:12:40Z")
Hello, I have a strange problem with my ELK setup. I'm using this docker container https://hub.docker.com/r/sebp/elk/ with the current version 651. So I'm using logstash version 6.5.1 I wrote a small script which copi… --- ## [How do I know if Elastic search is using logstash?](https://discuss.elastic.co/t/how-do-i-know-if-elastic-search-is-using-logstash/167009)
**Author:** [@gjfonte.ai](https://discuss.elastic.co/u/gjfonte.ai)\ **Replies:** 0\ **Last updated:** [February 4, 2019, 5:36pm UTC](https://discuss.elastic.co/t/how-do-i-know-if-elastic-search-is-using-logstash/167009 "2019-02-04T17:36:15Z")
I am using the Microsoft template preferred solution for Elastic search and I would like to know If I have got Logstash correctly configured. I have not set anything for the logstash config file during the installation. … --- ## [HTTP Input for Logstash](https://discuss.elastic.co/t/http-input-for-logstash/167078)
**Author:** [@Rahul\_Pm](https://discuss.elastic.co/u/Rahul_Pm)\ **Replies:** 0\ **Last updated:** [February 5, 2019, 8:55am UTC](https://discuss.elastic.co/t/http-input-for-logstash/167078 "2019-02-05T08:55:22Z")
Hi. I'm trying to push data to Logstash via http. The same worked well when I pushed the data to Elasticsearch. To connect to Elasticsearch, I used this URL : http://:9200/indexname/typename and it worked fine.For Logst… --- ## [Elastiscsearch Output - Attempted to send a bulk request](https://discuss.elastic.co/t/elastiscsearch-output-attempted-to-send-a-bulk-request/166703)
**Author:** [@Gochou](https://discuss.elastic.co/u/Gochou)\ **Replies:** 2\ **Last updated:** [February 5, 2019, 8:33am UTC](https://discuss.elastic.co/t/elastiscsearch-output-attempted-to-send-a-bulk-request/166703 "2019-02-05T08:33:56Z")
Hi, So from time to time I get this error in my logstash logs: \[2019-02-01T01:35:18,762\]\[ERROR\]\[logstash.outputs.elasticsearch\] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreach… --- ## [How do I set the proxy for logstash plugin install on windows?](https://discuss.elastic.co/t/how-do-i-set-the-proxy-for-logstash-plugin-install-on-windows/167041)
**Author:** [@Freebird](https://discuss.elastic.co/u/Freebird)\ **Replies:** 1\ **Last updated:** [February 5, 2019, 4:57am UTC](https://discuss.elastic.co/t/how-do-i-set-the-proxy-for-logstash-plugin-install-on-windows/167041 "2019-02-05T04:57:54Z")
I'm trying to install the elapsed plug-in and keep getting a connection error. Gooogling reveals that logstash doesn't use the browser proxy. The documentation indicates to 'export' the proxy - but that's a unix comman… --- ## [How to get field count in Elasticsearch filter plugin](https://discuss.elastic.co/t/how-to-get-field-count-in-elasticsearch-filter-plugin/167054)
**Author:** [@keep](https://discuss.elastic.co/u/keep)\ **Replies:** 0\ **Last updated:** [February 5, 2019, 4:35am UTC](https://discuss.elastic.co/t/how-to-get-field-count-in-elasticsearch-filter-plugin/167054 "2019-02-05T04:35:08Z")
Example : if \[type\] == "end" { elasticsearch { hosts =\> \["es-server"\] query =\> "select count(\*) as numCount from index" fields =\> { "numCount "} } result : numCount : nil --- ## [Suddenly unable to import parsed logs from logstash into elasticsearch](https://discuss.elastic.co/t/suddenly-unable-to-import-parsed-logs-from-logstash-into-elasticsearch/166439)
**Author:** [@cheem](https://discuss.elastic.co/u/cheem)\ **Replies:** 8\ **Last updated:** [February 4, 2019, 11:12pm UTC](https://discuss.elastic.co/t/suddenly-unable-to-import-parsed-logs-from-logstash-into-elasticsearch/166439 "2019-02-04T23:12:15Z")
About a day ago by now, my elasticsearch stack stopped accepting parsed nginx logs. Logstash and Elasticsearch are 6.5.4. Here are the filters for these logs: grok { match =\> {"message" =\> "%{IP:remote\_ip}?,?\\… --- ## [Prevent logstash from adding fields to json input](https://discuss.elastic.co/t/prevent-logstash-from-adding-fields-to-json-input/167008)
**Author:** [@reschenburg](https://discuss.elastic.co/u/reschenburg)\ **Replies:** 2\ **Last updated:** [February 4, 2019, 6:34pm UTC](https://discuss.elastic.co/t/prevent-logstash-from-adding-fields-to-json-input/167008 "2019-02-04T18:34:03Z")
Hi I've got a simple setup with a tcp input and kafka output. input { tcp { port =\> "${TCP\_PORT:1514}" codec =\> json } } output { stdout { codec =\> rubydebug } kafka { bootstrap\_servers =\> "${KAFKA… --- ## [Logstash-plugin: Data load takes thrice the time after upgrading from logstash 6.2.4 to 6.5.2 due to mutate filter](https://discuss.elastic.co/t/logstash-plugin-data-load-takes-thrice-the-time-after-upgrading-from-logstash-6-2-4-to-6-5-2-due-to-mutate-filter/166988)
**Author:** [@dynium](https://discuss.elastic.co/u/dynium)\ **Replies:** 0\ **Last updated:** [February 4, 2019, 4:09pm UTC](https://discuss.elastic.co/t/logstash-plugin-data-load-takes-thrice-the-time-after-upgrading-from-logstash-6-2-4-to-6-5-2-due-to-mutate-filter/166988 "2019-02-04T16:09:29Z")
I'm fetching data using the jdbc input filter for logstash and after some mutate converts and ruby filter calculations, the data is output to elasticsearch. My logstash config file for 6.2.4 is input { jdbc { jdbc\_… --- ## [Logstash putting quotes around nested json http output plugin](https://discuss.elastic.co/t/logstash-putting-quotes-around-nested-json-http-output-plugin/166985)
**Author:** [@wffpierce](https://discuss.elastic.co/u/wffpierce)\ **Replies:** 0\ **Last updated:** [February 4, 2019, 3:57pm UTC](https://discuss.elastic.co/t/logstash-putting-quotes-around-nested-json-http-output-plugin/166985 "2019-02-04T15:57:18Z")
So I am trying to use beats and logstash to ship to splunk as well as a number of other places. Splunk requires an "event" field to be the outer most of the json. I am having a very hard time doing this. Any help would b… --- ## [Impossible format date](https://discuss.elastic.co/t/impossible-format-date/166355)
**Author:** [@Viti](https://discuss.elastic.co/u/Viti)\ **Replies:** 13\ **Last updated:** [February 4, 2019, 3:50pm UTC](https://discuss.elastic.co/t/impossible-format-date/166355 "2019-02-04T15:50:58Z")
Hello, I need to format a date. The log line is this: 2019-01-29 12:59:51,8345358246519 ERROR - RGKAS055551 \* INFO \* 00003 \* method \* initSystem \* =====\> fail system down I have this: 2019-01-29 12:59:51,83453582465… --- ## [Extract value in nested json with array](https://discuss.elastic.co/t/extract-value-in-nested-json-with-array/166948)
**Author:** [@chibisuke](https://discuss.elastic.co/u/chibisuke)\ **Replies:** 3\ **Last updated:** [February 4, 2019, 3:02pm UTC](https://discuss.elastic.co/t/extract-value-in-nested-json-with-array/166948 "2019-02-04T15:02:35Z")
Dear community members, I can't get my head around how to access the following structure and retrieve one field to be able to aggregate later on it: "EM.simInfoValidation": \[{ "iccid": "898600D809149043559", "im… --- ## [Twitter logstash](https://discuss.elastic.co/t/twitter-logstash/166977)
**Author:** [@fedwe](https://discuss.elastic.co/u/fedwe)\ **Replies:** 0\ **Last updated:** [February 4, 2019, 2:54pm UTC](https://discuss.elastic.co/t/twitter-logstash/166977 "2019-02-04T14:54:45Z")
i am trying to make a twitter conf for the first time. It worked on words like thor hulk in keywords, but when i use like the keywords AlfaSports or stuff related to alfa telecom company nothing appears. and also if i u… --- ## [How to configure logstash-output-nagios, getting error Unknown setting 'nagios\_host' for nagios](https://discuss.elastic.co/t/how-to-configure-logstash-output-nagios-getting-error-unknown-setting-nagios-host-for-nagios/165776)
**Author:** [@abhijitnt](https://discuss.elastic.co/u/abhijitnt)\ **Replies:** 8\ **Last updated:** [February 4, 2019, 1:48pm UTC](https://discuss.elastic.co/t/how-to-configure-logstash-output-nagios-getting-error-unknown-setting-nagios-host-for-nagios/165776 "2019-02-04T13:48:53Z")
Hi All. I am trying to setup a Logstash Nagios plugin which can direct send logstash file errors to Nagios server. Here is what I have done so far. Its failing on logstash test config. Any help is appreciated on this. $… --- ## [Kafka output topic\_id](https://discuss.elastic.co/t/kafka-output-topic-id/166756)
**Author:** [@Reb](https://discuss.elastic.co/u/Reb)\ **Replies:** 1\ **Last updated:** [February 4, 2019, 12:57pm UTC](https://discuss.elastic.co/t/kafka-output-topic-id/166756 "2019-02-04T12:57:04Z")
I try to use this plugin output { kafka{ bootstrap\_servers =\> "dal-b-23971:29090,dal-b-23971:29091,dal-b-23971:29092" topic\_id =\> " \*\*\_%{\[fields\]\[app\_id\]}\_\*\* " … --- ## [Multiple matches from one line](https://discuss.elastic.co/t/multiple-matches-from-one-line/166851)
**Author:** [@epacke](https://discuss.elastic.co/u/epacke)\ **Replies:** 6\ **Last updated:** [February 4, 2019, 12:50pm UTC](https://discuss.elastic.co/t/multiple-matches-from-one-line/166851 "2019-02-04T12:50:41Z")
Hi! I am struggling with syslog from one of our vendors (F5). When something goes down a syslog message is sent with the thing that has failed together with the last states of the monitor. Two sample log lines could lo… --- ## [Parse mixed type of data inside json {\[array\],keys:values}](https://discuss.elastic.co/t/parse-mixed-type-of-data-inside-json-array-keys-values/166590)
**Author:** [@wonderg](https://discuss.elastic.co/u/wonderg)\ **Replies:** 2\ **Last updated:** [February 4, 2019, 12:26pm UTC](https://discuss.elastic.co/t/parse-mixed-type-of-data-inside-json-array-keys-values/166590 "2019-02-04T12:26:19Z")
How to parse mixed type of data inside json {\[array\],keys:values} ? { "body": { "date": "1548930893906", "id": "3987741004", "origin": "web", "parties": \[{ "direction": "in", "from": { "name": "Name", "number":… --- ## [Logstash multiple pipelines bug?](https://discuss.elastic.co/t/logstash-multiple-pipelines-bug/166582)
**Author:** [@sabdoul](https://discuss.elastic.co/u/sabdoul)\ **Replies:** 3\ **Last updated:** [February 4, 2019, 10:31am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-bug/166582 "2019-02-04T10:31:05Z")
Hello, I made a mutiple pipeline with version 6.6 of logstash. But, he has behavior that is unexpected. Indeed I have my multiple pipeline that generates duplicates of the data as if making a loop before stopping the p… --- ## [\[SOLVED\] Increment field value in logstash filter](https://discuss.elastic.co/t/solved-increment-field-value-in-logstash-filter/166757)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 3\ **Last updated:** [February 4, 2019, 9:42am UTC](https://discuss.elastic.co/t/solved-increment-field-value-in-logstash-filter/166757 "2019-02-04T09:42:29Z")
Hello, I have a data taken from the filter elasticsearch query. elasticsearch { hosts =\> \["localhost:9200"\] index =\> "elasticinput" query =\> '\_id=%{fingerprint}' fields =\> { "request\_ip\_count" =\> "new\_request\_ip\_c… --- ## [Monitoring dead letter queue (DLQ)](https://discuss.elastic.co/t/monitoring-dead-letter-queue-dlq/166715)
**Author:** [@beckjim](https://discuss.elastic.co/u/beckjim)\ **Replies:** 2\ **Last updated:** [February 4, 2019, 8:17am UTC](https://discuss.elastic.co/t/monitoring-dead-letter-queue-dlq/166715 "2019-02-04T08:17:59Z")
Hello all, I'm wondering is there a proper way to monitor the contents of the dead letter queue? Maybe I missed something ... None of the monitoring APIs seem to yield are data about dlq's. Items of interesst would b… --- ## [Pipeline starts before the compilation is completed with Java execution](https://discuss.elastic.co/t/pipeline-starts-before-the-compilation-is-completed-with-java-execution/166817)
**Author:** [@KBuev](https://discuss.elastic.co/u/KBuev)\ **Replies:** 2\ **Last updated:** [February 4, 2019, 5:57am UTC](https://discuss.elastic.co/t/pipeline-starts-before-the-compilation-is-completed-with-java-execution/166817 "2019-02-04T05:57:13Z")
Hi, everyone. I've been trying out the Java execution engine in Logstash 6.6.0 since it's now considered GA. We have several Logstash instances running the same pipeline with Persistent Queue enabled. Sometimes we need… --- ## [Parse mix of String, JSON and KeyValue](https://discuss.elastic.co/t/parse-mix-of-string-json-and-keyvalue/164791)
**Author:** [@Mohammed\_Farhan\_Ejaz](https://discuss.elastic.co/u/Mohammed_Farhan_Ejaz)\ **Replies:** 2\ **Last updated:** [February 4, 2019, 5:49am UTC](https://discuss.elastic.co/t/parse-mix-of-string-json-and-keyvalue/164791 "2019-02-04T05:49:59Z")
2019-01-18T11:33:40,011 WXXX11100111L 43992 CM http-nio-8080-exec-8 MY-TEST POST configurations {a=b} {c=d, x=y} { "x": "abc", "country":"india", "lang":"EN", "project":"my-project"} 200 4266 I have the above log entry … --- ## [Logstash 6.6 - Cannot install logstash-filter-geoip plugin :ERROR](https://discuss.elastic.co/t/logstash-6-6-cannot-install-logstash-filter-geoip-plugin-error/166659)
**Author:** [@taku333](https://discuss.elastic.co/u/taku333)\ **Replies:** 6\ **Last updated:** [February 4, 2019, 2:48am UTC](https://discuss.elastic.co/t/logstash-6-6-cannot-install-logstash-filter-geoip-plugin-error/166659 "2019-02-04T02:48:42Z")
I attempted to install the logstash-filter-geoip plugin via proxy. But installed Faild. I set and run setting http\_proxy and https\_proxy,but still it did not go well.. Is there something wrong with the procedure? ◆L… --- ## [Date difference in events received from filebeat](https://discuss.elastic.co/t/date-difference-in-events-received-from-filebeat/166866)
**Author:** [@skarcos](https://discuss.elastic.co/u/skarcos)\ **Replies:** 2\ **Last updated:** [February 3, 2019, 7:26pm UTC](https://discuss.elastic.co/t/date-difference-in-events-received-from-filebeat/166866 "2019-02-03T19:26:51Z")
Hello, I am using filebeat on a remote linux machine to send events to logstash. I am using grok and the date module to get events' timestamps. Here is my complete configuration: input { beats { type =\> "gemma\_… --- ## [Setting \`congestion\_threshold\` in plugin \`beats\` --getting below error any solution to fix this](https://discuss.elastic.co/t/setting-congestion-threshold-in-plugin-beats-getting-below-error-any-solution-to-fix-this/166853)
**Author:** [@rajesh123vit](https://discuss.elastic.co/u/rajesh123vit)\ **Replies:** 3\ **Last updated:** [February 3, 2019, 5:29pm UTC](https://discuss.elastic.co/t/setting-congestion-threshold-in-plugin-beats-getting-below-error-any-solution-to-fix-this/166853 "2019-02-03T17:29:41Z")
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties \[2019-02-03T11:55:15,766\]\[FATAL\]\[logstash.runn… --- ## [File input not read logstah \[SOLVED\]](https://discuss.elastic.co/t/file-input-not-read-logstah-solved/166829)
**Author:** [@Darkside](https://discuss.elastic.co/u/Darkside)\ **Replies:** 11\ **Last updated:** [February 3, 2019, 3:58pm UTC](https://discuss.elastic.co/t/file-input-not-read-logstah-solved/166829 "2019-02-03T15:58:48Z")
Hi i am new to ELK i have some troubles with reading file. My conf file input { file { path =\> "/files/nginx\_logs" start\_position =\> "beginning" ignore\_older =\> 0 } } output { stdout { } } a sample of log ngi… --- ## [Copy last uuid value in the configuration](https://discuss.elastic.co/t/copy-last-uuid-value-in-the-configuration/166825)
**Author:** [@et159](https://discuss.elastic.co/u/et159)\ **Replies:** 3\ **Last updated:** [February 2, 2019, 4:42pm UTC](https://discuss.elastic.co/t/copy-last-uuid-value-in-the-configuration/166825 "2019-02-02T16:42:02Z")
Hello, I'm new to ELK , I have a grok filter to parse logs , I added tags as process started and process ended to calculate the session length for each connection, but the session had no session ID , is i generated uui… --- ## [Can I use filter inside filter?](https://discuss.elastic.co/t/can-i-use-filter-inside-filter/163259)
**Author:** [@et159](https://discuss.elastic.co/u/et159)\ **Replies:** 8\ **Last updated:** [February 2, 2019, 4:15pm UTC](https://discuss.elastic.co/t/can-i-use-filter-inside-filter/163259 "2019-02-02T16:15:57Z")
I had a logstash configuration as below : input { 3 input sources } filter { if \[field\] \[source\]== "source1" { do .... } else \[field\] \[source\]== "source2" { do .... } else… --- ## [Curl a webservice using ruby filter](https://discuss.elastic.co/t/curl-a-webservice-using-ruby-filter/166734)
**Author:** [@ruby.gem](https://discuss.elastic.co/u/ruby.gem)\ **Replies:** 3\ **Last updated:** [February 1, 2019, 10:39pm UTC](https://discuss.elastic.co/t/curl-a-webservice-using-ruby-filter/166734 "2019-02-01T22:39:50Z")
Hello, Im trying to consume some events from a webservice with curl using ruby code in logstash filter. The service requires an ip address to be supplied with -d switch. curl -s -XPOST -d 'ip=151.101.0.81' 'http://api… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=605) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=607)