# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=817 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 818 --- ## [S3 output retry interval](https://discuss.elastic.co/t/s3-output-retry-interval/120751)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 0\ **Last updated:** [February 21, 2018, 2:01am UTC](https://discuss.elastic.co/t/s3-output-retry-interval/120751 "2018-02-21T02:01:24Z")
Hi, Does anyone know how to adjust the retry interval on the s3 output plugin when the connection fails? Default seems to be to retry ever 2 seconds or so but I want to change this to every 30 minutes. There doesn't ap… --- ## [Add nested document from another index](https://discuss.elastic.co/t/add-nested-document-from-another-index/120744)
**Author:** [@Baljeet\_Dogra](https://discuss.elastic.co/u/Baljeet_Dogra)\ **Replies:** 0\ **Last updated:** [February 21, 2018, 12:14am UTC](https://discuss.elastic.co/t/add-nested-document-from-another-index/120744 "2018-02-21T00:14:57Z")
I have a requirement to lookup another index in elasticsearch and when the document is found I need to embed the entire document in another document in another index. employee-index (assume already populated) { \_id: em… --- ## [Append field if document\_id already exists](https://discuss.elastic.co/t/append-field-if-document-id-already-exists/120742)
**Author:** [@Baljeet\_Dogra](https://discuss.elastic.co/u/Baljeet_Dogra)\ **Replies:** 0\ **Last updated:** [February 20, 2018, 11:55pm UTC](https://discuss.elastic.co/t/append-field-if-document-id-already-exists/120742 "2018-02-20T23:55:47Z")
I have a csv file with multiple records for each document where the data is partially different. doc\_id insurance\_policy policy\_holder C12 P-C12-2017 Mi… --- ## [Please help - Error in logstash shipper](https://discuss.elastic.co/t/please-help-error-in-logstash-shipper/120058)
**Author:** [@pklanka](https://discuss.elastic.co/u/pklanka)\ **Replies:** 2\ **Last updated:** [February 20, 2018, 10:33pm UTC](https://discuss.elastic.co/t/please-help-error-in-logstash-shipper/120058 "2018-02-20T22:33:17Z")
Hi there We have been facing some challenges with logstash shipper - which is basically showing the below error message millions of times. We do not see any logs processed. Certificates are not expired. Can anyone plea… --- ## [Logstash JVM heap and GC issues](https://discuss.elastic.co/t/logstash-jvm-heap-and-gc-issues/120711)
**Author:** [@gorizon](https://discuss.elastic.co/u/gorizon)\ **Replies:** 1\ **Last updated:** [February 20, 2018, 10:30pm UTC](https://discuss.elastic.co/t/logstash-jvm-heap-and-gc-issues/120711 "2018-02-20T22:30:42Z")
Hi I've got an issue with my JVM heap steadily increasing until all the space is used up. Seems that the garbage collection is removing less and less space until eventually the heap is used up. Here is a picture of wha… --- ## [Getting "Invalid Frame Type, received: 1" on 6.2.1 stack](https://discuss.elastic.co/t/getting-invalid-frame-type-received-1-on-6-2-1-stack/120378)
**Author:** [@ralfr](https://discuss.elastic.co/u/ralfr)\ **Replies:** 1\ **Last updated:** [February 20, 2018, 8:55pm UTC](https://discuss.elastic.co/t/getting-invalid-frame-type-received-1-on-6-2-1-stack/120378 "2018-02-20T20:55:14Z")
I'm running out of ideas. Having the 6.2.1 stack installed (ES, Logstash, Kibana and X-Pack). As soon, as I start Filebeat on a host in order to forward syslog messages to the ELK stack, I do get Caused by: org.logstash… --- ## [Convert decimal string IP to ipv4 dotted decimal](https://discuss.elastic.co/t/convert-decimal-string-ip-to-ipv4-dotted-decimal/120476)
**Author:** [@felixon](https://discuss.elastic.co/u/felixon)\ **Replies:** 1\ **Last updated:** [February 20, 2018, 8:26pm UTC](https://discuss.elastic.co/t/convert-decimal-string-ip-to-ipv4-dotted-decimal/120476 "2018-02-20T20:26:28Z")
Hi, I am sending JSON formatted logs to my 5.4 Logstash. I am using Json Filter Plugin. One of the JSON fields is Source\_IP, the point is that Source\_IP field is coming as decimal long number like "3105947710" (which… --- ## [Index template error](https://discuss.elastic.co/t/index-template-error/120291)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 1\ **Last updated:** [February 20, 2018, 8:22pm UTC](https://discuss.elastic.co/t/index-template-error/120291 "2018-02-20T20:22:23Z")
Getting an odd error when attempting to import an index template. Can someone point out where my syntax is wrong? The referenced field appears to be the same as its siblings. Error message: { "error": { "root\_caus… --- ## [Can I grok out the host field if its not a valid IP? Also, what is the host field?](https://discuss.elastic.co/t/can-i-grok-out-the-host-field-if-its-not-a-valid-ip-also-what-is-the-host-field/120515)
**Author:** [@red888](https://discuss.elastic.co/u/red888)\ **Replies:** 1\ **Last updated:** [February 20, 2018, 8:19pm UTC](https://discuss.elastic.co/t/can-i-grok-out-the-host-field-if-its-not-a-valid-ip-also-what-is-the-host-field/120515 "2018-02-20T20:19:57Z")
Not sure how to handle this. I have this in my index template: "host": { "type": "IP" }, In logstash I have this: input { tcp { type =\> "weblogs" port =\> "9999" } } Now on a production server "host" is always… --- ## [Wrong indexing of Documents using ELK](https://discuss.elastic.co/t/wrong-indexing-of-documents-using-elk/120696)
**Author:** [@SRIpandu1729](https://discuss.elastic.co/u/SRIpandu1729)\ **Replies:** 3\ **Last updated:** [February 20, 2018, 7:18pm UTC](https://discuss.elastic.co/t/wrong-indexing-of-documents-using-elk/120696 "2018-02-20T19:18:16Z")
Whenever I am trying to index log data, some of the log data is being wrongly indexed into a new index. The logstash configuration file is as follows: input { file { path =\> "/home/user/DATA/\*.log" … --- ## [Newbie - Logstash appears to be losing logs maybe cause by my Filter](https://discuss.elastic.co/t/newbie-logstash-appears-to-be-losing-logs-maybe-cause-by-my-filter/120714)
**Author:** [@hassanbabaie](https://discuss.elastic.co/u/hassanbabaie)\ **Replies:** 0\ **Last updated:** [February 20, 2018, 7:08pm UTC](https://discuss.elastic.co/t/newbie-logstash-appears-to-be-losing-logs-maybe-cause-by-my-filter/120714 "2018-02-20T19:08:17Z")
My setup: Azure NSG Flowlogs -\> Azure Storage -\> Logstash with Azure Plugin -\> Graylog As far as I can tell the Azure Logstash plugin is pulling all the logs however I think my filter is then causing some of these to b… --- ## [Unable to convert timestamp to Date field](https://discuss.elastic.co/t/unable-to-convert-timestamp-to-date-field/120098)
**Author:** [@Karanushah.1992](https://discuss.elastic.co/u/Karanushah.1992)\ **Replies:** 12\ **Last updated:** [February 20, 2018, 6:44pm UTC](https://discuss.elastic.co/t/unable-to-convert-timestamp-to-date-field/120098 "2018-02-20T18:44:30Z")
I am trying to convert a string field called timestamp with value 1510722000000 in date format field in Logstash. My sole purpose is to visualize data in kibana using this date field. I tried using the date filter but it… --- ## [XML filter - Not parsing xmls with version tag](https://discuss.elastic.co/t/xml-filter-not-parsing-xmls-with-version-tag/120499)
**Author:** [@tag\_v](https://discuss.elastic.co/u/tag_v)\ **Replies:** 4\ **Last updated:** [February 20, 2018, 6:41pm UTC](https://discuss.elastic.co/t/xml-filter-not-parsing-xmls-with-version-tag/120499 "2018-02-20T18:41:09Z")
XML filter not able to parse message if it contains xml version. Logstash able to parse xml perfectly with message: \\ \ **Author:** [@kabali12345](https://discuss.elastic.co/u/kabali12345)\ **Replies:** 1\ **Last updated:** [February 20, 2018, 5:48pm UTC](https://discuss.elastic.co/t/problem-with-logstash-http-poller-plugin/120232 "2018-02-20T17:48:02Z") Hi all here i am trying to call rest API with http\_poller plugin. Here is my logstash conf input { http\_poller { urls =\> { test2 =\> { method =\> get user =\> "XXXX" password =\> "XXXX" … --- ## [Logstash works for a few minutes then stops](https://discuss.elastic.co/t/logstash-works-for-a-few-minutes-then-stops/120268)
**Author:** [@gorizon](https://discuss.elastic.co/u/gorizon)\ **Replies:** 8\ **Last updated:** [February 20, 2018, 5:33pm UTC](https://discuss.elastic.co/t/logstash-works-for-a-few-minutes-then-stops/120268 "2018-02-20T17:33:08Z")
This is a new instance for testing purposes in a personal lab. After setting up logstash and adding configuration I start the service. Logstash proceeds to run for a minute or two and just completely stops either recei… --- ## [Filtering Fields whose values are negative](https://discuss.elastic.co/t/filtering-fields-whose-values-are-negative/120688)
**Author:** [@govinda.rao](https://discuss.elastic.co/u/govinda.rao)\ **Replies:** 0\ **Last updated:** [February 20, 2018, 4:58pm UTC](https://discuss.elastic.co/t/filtering-fields-whose-values-are-negative/120688 "2018-02-20T16:58:29Z")
Hi, I need a help on filtering the negative values in below logstash IF condition. Below code calculates the elapsed time. Some times i used to get the negative values for elapsed\_time. How can i filter those negative v… --- ## [Logstash - change existing date format](https://discuss.elastic.co/t/logstash-change-existing-date-format/120683)
**Author:** [@Aleksander\_de](https://discuss.elastic.co/u/Aleksander_de)\ **Replies:** 0\ **Last updated:** [February 20, 2018, 4:32pm UTC](https://discuss.elastic.co/t/logstash-change-existing-date-format/120683 "2018-02-20T16:32:23Z")
I need to change the date format in logstash from field "DATE" -\> 10/21/2017 5:26:42 PM to 2017-10-21T17:26:42.000Z. I don't mean to just change the mapping in that field (which I did) but to change existing data. Date … --- ## [Aggregate filter plugin](https://discuss.elastic.co/t/aggregate-filter-plugin/119632)
**Author:** [@scch](https://discuss.elastic.co/u/scch)\ **Replies:** 10\ **Last updated:** [February 20, 2018, 3:49pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin/119632 "2018-02-20T15:49:10Z")
occurred\_on Bytes\_Out user\_id 2017-12-31T10:25:49.000-0600 1718 ts2341 2017-12-31T10:25:08.000-0600 1070 ts2341 2017-12-31T10:25:08.000-0600 355 ts2341 2017-12-29T16:55:01.000-0600 255 gs2121 2017-12-29T1… --- ## [Logstash Filter Elasticsearch - Referencing Fields](https://discuss.elastic.co/t/logstash-filter-elasticsearch-referencing-fields/120070)
**Author:** [@KRasekhi](https://discuss.elastic.co/u/KRasekhi)\ **Replies:** 1\ **Last updated:** [February 20, 2018, 3:00pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-referencing-fields/120070 "2018-02-20T15:00:44Z")
Hi, I am a bit confused on reference some fields , for example - fields =\> { "host" =\> "last\_logon\_host" "@timestamp" =\> "last\_logon\_time" "event\_data.LogonType" =\> "last\_logon\_type" } all of them work except ev… --- ## [Accesing with Logstash to Filebeat parameter fields](https://discuss.elastic.co/t/accesing-with-logstash-to-filebeat-parameter-fields/120571)
**Author:** [@Asier\_Saluena](https://discuss.elastic.co/u/Asier_Saluena)\ **Replies:** 1\ **Last updated:** [February 20, 2018, 1:35pm UTC](https://discuss.elastic.co/t/accesing-with-logstash-to-filebeat-parameter-fields/120571 "2018-02-20T13:35:14Z")
Hi all, I'm trying new ELK version (6.2), and I'm using the new config options 'fields' to create a field named logtype. I want to detect logs in Logstash using this parameter. filebeat.prospectors: - type: log enab… --- ## [IS ELK 6.x supported in AIX?](https://discuss.elastic.co/t/is-elk-6-x-supported-in-aix/120626)
**Author:** [@Rahulkumar](https://discuss.elastic.co/u/Rahulkumar)\ **Replies:** 8\ **Last updated:** [February 20, 2018, 1:34pm UTC](https://discuss.elastic.co/t/is-elk-6-x-supported-in-aix/120626 "2018-02-20T13:34:26Z")
I want to use few of the beats and logstash on AIX system. Want to know, if AIX supports latest versions? --- ## [Logstash 6.2.1 , jdk9 - Unsupported platform: x86\_64-windows](https://discuss.elastic.co/t/logstash-6-2-1-jdk9-unsupported-platform-x86-64-windows/120213)
**Author:** [@jmota](https://discuss.elastic.co/u/jmota)\ **Replies:** 6\ **Last updated:** [February 20, 2018, 1:18pm UTC](https://discuss.elastic.co/t/logstash-6-2-1-jdk9-unsupported-platform-x86-64-windows/120213 "2018-02-20T13:18:42Z")
confirmed tat 6.2.1 works with oracle j9 but i m getting unsupported platform- thxs for any help. https://www.elastic.co/support/matrix#matrix\_jvm C:\\Kibana\_ElasticSearch\\logstash-6.2.1\>bin/logstash -f my\_logstash-sim… --- ## [Error regarding logstash pipeline has been started but index is not being created](https://discuss.elastic.co/t/error-regarding-logstash-pipeline-has-been-started-but-index-is-not-being-created/119857)
**Author:** [@pratibha1](https://discuss.elastic.co/u/pratibha1)\ **Replies:** 1\ **Last updated:** [February 20, 2018, 12:27pm UTC](https://discuss.elastic.co/t/error-regarding-logstash-pipeline-has-been-started-but-index-is-not-being-created/119857 "2018-02-20T12:27:52Z")
C:\\Users\\pratiswa\\Downloads\\logstash-6.2.1\\logstash-6.2.1\\bin\>logstash -f C:\\Users\\pratiswa\\Desktop\\logstash\_nba.config Sending Logstash's logs to C:/Users/pratiswa/Downloads/logstash-6.2.1/logstash-6.2.1/logs which is … --- ## [Logstash HTTP input 403 Response](https://discuss.elastic.co/t/logstash-http-input-403-response/120611)
**Author:** [@Daniel\_Tebernum](https://discuss.elastic.co/u/Daniel_Tebernum)\ **Replies:** 0\ **Last updated:** [February 20, 2018, 11:05am UTC](https://discuss.elastic.co/t/logstash-http-input-403-response/120611 "2018-02-20T11:05:36Z")
Hello, I've read through a lot of documentation and topics here, but I still have problems sending a JSON document to logstash. I have two Docker container. One is for my nodejs application, which is sending JSON docum… --- ## [Does 5.6.X support Logstash Multiple pipelining](https://discuss.elastic.co/t/does-5-6-x-support-logstash-multiple-pipelining/120603)
**Author:** [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Replies:** 1\ **Last updated:** [February 20, 2018, 10:39am UTC](https://discuss.elastic.co/t/does-5-6-x-support-logstash-multiple-pipelining/120603 "2018-02-20T10:39:50Z")
team, Could you please let me know that, I can achieve Logstash muliple pipelines feature using Logstash 5.6.2 ? Thanks. --- ## [Logstash Grok - duplicate entries after parsing](https://discuss.elastic.co/t/logstash-grok-duplicate-entries-after-parsing/120563)
**Author:** [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Replies:** 3\ **Last updated:** [February 20, 2018, 10:36am UTC](https://discuss.elastic.co/t/logstash-grok-duplicate-entries-after-parsing/120563 "2018-02-20T10:36:26Z")
I have a peculiar situation where , the grok match parsing is putting duplicate entries after parsing. I have tried break\_on\_match =\> true ( with true -\> it is not able to parse the required fields), on false, it is put… --- ## [Netflow sourceid](https://discuss.elastic.co/t/netflow-sourceid/120605)
**Author:** [@jeppef](https://discuss.elastic.co/u/jeppef)\ **Replies:** 0\ **Last updated:** [February 20, 2018, 10:33am UTC](https://discuss.elastic.co/t/netflow-sourceid/120605 "2018-02-20T10:33:51Z")
We collect netflow from a node with several source id's where we want to seperate these based on the id, is there a way to extract the source id field ? similar to the include\_flowset\_id --- ## [Logstash confluent avro topic read](https://discuss.elastic.co/t/logstash-confluent-avro-topic-read/120601)
**Author:** [@Shekhar\_Pandey](https://discuss.elastic.co/u/Shekhar_Pandey)\ **Replies:** 0\ **Last updated:** [February 20, 2018, 10:09am UTC](https://discuss.elastic.co/t/logstash-confluent-avro-topic-read/120601 "2018-02-20T10:09:00Z")
Hi...I create a Java producer using avro in kafka topic....but when I try to read the topic using logstash...its giving empty messages....my confi file is given below... input { kafka { bootstrap\_servers =\> "localhost… --- ## [Digest Authentication in logstash input plugins](https://discuss.elastic.co/t/digest-authentication-in-logstash-input-plugins/120587)
**Author:** [@shahrukhkhan](https://discuss.elastic.co/u/shahrukhkhan)\ **Replies:** 0\ **Last updated:** [February 20, 2018, 9:42am UTC](https://discuss.elastic.co/t/digest-authentication-in-logstash-input-plugins/120587 "2018-02-20T09:42:32Z")
Hi Team, So i have a requirement where i have to input data in logstash through an http api. I am aware that a plugin called "http-poller" exists for that. But my question is whether this plugin or any other plugin whi… --- ## [Capture response from HTTP get requests using HTTP Output Plugin in Logstash](https://discuss.elastic.co/t/capture-response-from-http-get-requests-using-http-output-plugin-in-logstash/120560)
**Author:** [@davidgong](https://discuss.elastic.co/u/davidgong)\ **Replies:** 0\ **Last updated:** [February 20, 2018, 6:06am UTC](https://discuss.elastic.co/t/capture-response-from-http-get-requests-using-http-output-plugin-in-logstash/120560 "2018-02-20T06:06:51Z")
Hi ELK support, I am able to call the below web API successfully (verified through the application log), however, no response was captured on the console. Is there a way for me to capture the http response via a console… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=816) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=818)