# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1025 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1026 --- ## [Dealing with varied log format](https://discuss.elastic.co/t/dealing-with-varied-log-format/77324)
**Author:** [@kristoffer](https://discuss.elastic.co/u/kristoffer)\ **Replies:** 2\ **Last updated:** [March 5, 2017, 6:15am UTC](https://discuss.elastic.co/t/dealing-with-varied-log-format/77324 "2017-03-05T06:15:29Z")
Hello! I just finished reading the Logstash Book (great introduction to the subject!) and would like to start experimenting with proper log collection in my environment. I would like to start off with out firewall. Get… --- ## [How to pass JVM parameters to Logstash?](https://discuss.elastic.co/t/how-to-pass-jvm-parameters-to-logstash/77406)
**Author:** [@Ranieri\_Mazili\_de\_Ol](https://discuss.elastic.co/u/Ranieri_Mazili_de_Ol)\ **Replies:** 1\ **Last updated:** [March 5, 2017, 3:57am UTC](https://discuss.elastic.co/t/how-to-pass-jvm-parameters-to-logstash/77406 "2017-03-05T03:57:36Z")
Hello, I need to pass additional JVM parameters to logstash when starting. I've tried modify the jvm.options that's inside conf directory but it haven't any effect. I could make it work adding the parameters changing s… --- ## [Logstash date filter not replacing @timestamp field](https://discuss.elastic.co/t/logstash-date-filter-not-replacing-timestamp-field/75549)
**Author:** [@Shirel\_Vaiman](https://discuss.elastic.co/u/Shirel_Vaiman)\ **Replies:** 3\ **Last updated:** [March 4, 2017, 9:26pm UTC](https://discuss.elastic.co/t/logstash-date-filter-not-replacing-timestamp-field/75549 "2017-03-04T21:26:39Z")
Hi all, Please assist, as I'm already becoming desperate. I'm unable to replace the @timestamp field with the actual log time. I've tried all kinds of help from the internet but no luck. This is my logstash filter date c… --- ## [INFO Error publishing events (retrying): read tcp read: connection reset by peer](https://discuss.elastic.co/t/info-error-publishing-events-retrying-read-tcp-read-connection-reset-by-peer/77358)
**Author:** [@wmstein](https://discuss.elastic.co/u/wmstein)\ **Replies:** 3\ **Last updated:** [March 4, 2017, 5:44pm UTC](https://discuss.elastic.co/t/info-error-publishing-events-retrying-read-tcp-read-connection-reset-by-peer/77358 "2017-03-04T17:44:18Z")
I put filebeat on a prod box and can not get filebeat to push to Logstash on my elkserver box. I am getting the below error from the filebeat.log file INFO Error publishing events (retrying): read tcp IP:45790-\>ELK\_IP:… --- ## [Attempted to resurrect connection to dead ES instance - but wrong host?](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance-but-wrong-host/77282)
**Author:** [@guenther](https://discuss.elastic.co/u/guenther)\ **Replies:** 1\ **Last updated:** [March 4, 2017, 2:50pm UTC](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance-but-wrong-host/77282 "2017-03-04T14:50:53Z")
Hi, we are using logstash (version 5.2.2) and ES (version 5.2.2) in a docker swarm. The logstash instance is only for metricbeats and works very well. input { beats { port =\> "5044" } } filter { } output… --- ## [S3 input silently fails if no trailing slash](https://discuss.elastic.co/t/s3-input-silently-fails-if-no-trailing-slash/77388)
**Author:** [@yomateo](https://discuss.elastic.co/u/yomateo)\ **Replies:** 0\ **Last updated:** [March 4, 2017, 2:26pm UTC](https://discuss.elastic.co/t/s3-input-silently-fails-if-no-trailing-slash/77388 "2017-03-04T14:26:17Z")
Heads up.. If you're using the s3 input and you don't see any action in your logs it may likely be the case that the (current) version requires a trailing slash if you're wanting to parse a directory. Ommitting a tr… --- ## [Encoding and logstash](https://discuss.elastic.co/t/encoding-and-logstash/77374)
**Author:** [@logman](https://discuss.elastic.co/u/logman)\ **Replies:** 0\ **Last updated:** [March 4, 2017, 5:11am UTC](https://discuss.elastic.co/t/encoding-and-logstash/77374 "2017-03-04T05:11:21Z")
We frequently have documents fail due to encoding issues. A typical error message is {:timestamp=\>"2017-03-03T09:04:54.473000+0000", :message=\>"Received an event that has a different character encoding than you config… --- ## [How to set my own field value as @timestamp](https://discuss.elastic.co/t/how-to-set-my-own-field-value-as-timestamp/77366)
**Author:** [@sharon.c](https://discuss.elastic.co/u/sharon.c)\ **Replies:** 1\ **Last updated:** [March 3, 2017, 11:04pm UTC](https://discuss.elastic.co/t/how-to-set-my-own-field-value-as-timestamp/77366 "2017-03-03T23:04:28Z")
I am porting data from Mysql to Elasticsearch using logtash 5.2 I need to have a field named record\_time as the timestamp in Elasticsearch, I used date filter, and it does not work, and there is no warning. input { j… --- ## [Using message pattern match to shut down Logstash](https://discuss.elastic.co/t/using-message-pattern-match-to-shut-down-logstash/77367)
**Author:** [@john6](https://discuss.elastic.co/u/john6)\ **Replies:** 0\ **Last updated:** [March 3, 2017, 10:04pm UTC](https://discuss.elastic.co/t/using-message-pattern-match-to-shut-down-logstash/77367 "2017-03-03T22:04:40Z")
What would be the best way of gracefully shutting down Logstash based on a message pattern match? The pattern match would indicate the file input stream is dead. if (\[message\] =~ "patternA") { ruby{ code … --- ## [Installing my own Plugin - HELP! :-)](https://discuss.elastic.co/t/installing-my-own-plugin-help/77192)
**Author:** [@dcarissi](https://discuss.elastic.co/u/dcarissi)\ **Replies:** 0\ **Last updated:** [March 2, 2017, 5:21pm UTC](https://discuss.elastic.co/t/installing-my-own-plugin-help/77192 "2017-03-02T17:21:32Z")
Hi, I'm using Logstash version 2.4.1 and am trying to create and install my own plugin. Back in Logstash version 1.5.4, I created a gem file and then used the plugin installer and it would appear in the list of plugins.… --- ## [Logstash.pipeline error: A plugin had an unrecoverable error](https://discuss.elastic.co/t/logstash-pipeline-error-a-plugin-had-an-unrecoverable-error/77228)
**Author:** [@wmstein](https://discuss.elastic.co/u/wmstein)\ **Replies:** 3\ **Last updated:** [March 3, 2017, 6:00pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-a-plugin-had-an-unrecoverable-error/77228 "2017-03-03T18:00:04Z")
I am setting up a demo for our CEO to review - Kibana dasboard. I have this installed on a local server and have filebeat installed on that elk server to create logs for the Kibana demo. It was working yesterday and I ha… --- ## [Error with parsing Azure json multiline files](https://discuss.elastic.co/t/error-with-parsing-azure-json-multiline-files/77307)
**Author:** [@111148](https://discuss.elastic.co/u/111148)\ **Replies:** 7\ **Last updated:** [March 3, 2017, 2:20pm UTC](https://discuss.elastic.co/t/error-with-parsing-azure-json-multiline-files/77307 "2017-03-03T14:20:32Z")
Hello, Please advise how to solve this. Here is my logstash config file: --- ## [Logstash 5.x ruby hash and mutate filter are crashing or not working](https://discuss.elastic.co/t/logstash-5-x-ruby-hash-and-mutate-filter-are-crashing-or-not-working/77140)
**Author:** [@Kuckkuck](https://discuss.elastic.co/u/Kuckkuck)\ **Replies:** 3\ **Last updated:** [March 3, 2017, 10:42am UTC](https://discuss.elastic.co/t/logstash-5-x-ruby-hash-and-mutate-filter-are-crashing-or-not-working/77140 "2017-03-03T10:42:23Z")
Hi, I have problems converting my settings from logstash 2.x to logstash 5.2. My log entry logs like this: "@timestamp" =\> 2017-03-02T12:56:01.946Z, "@version" =\> "1", "index\_date" =\> "2017-03-02", "message" … --- ## [Logstash never stops](https://discuss.elastic.co/t/logstash-never-stops/77170)
**Author:** [@debarun](https://discuss.elastic.co/u/debarun)\ **Replies:** 1\ **Last updated:** [March 3, 2017, 9:50am UTC](https://discuss.elastic.co/t/logstash-never-stops/77170 "2017-03-03T09:50:37Z")
Hi, When I am reading a file with logstash, the logstash reads the file once and then goes back again to the top to read it again. I am using the below config file:- input { file { path =\> "/apps/… --- ## [Can you grok a grok/aggregated result?](https://discuss.elastic.co/t/can-you-grok-a-grok-aggregated-result/76742)
**Author:** [@MaciejM](https://discuss.elastic.co/u/MaciejM)\ **Replies:** 1\ **Last updated:** [March 3, 2017, 9:27am UTC](https://discuss.elastic.co/t/can-you-grok-a-grok-aggregated-result/76742 "2017-03-03T09:27:24Z")
Hi Everyone, I'm working on a categorization of results based on what is in the logs. I thought that if I aggregated the categories from the file and matched on those I could achieve the desired result. Could someone p… --- ## [Problem with logstash config](https://discuss.elastic.co/t/problem-with-logstash-config/77275)
**Author:** [@Artyom\_Davydov](https://discuss.elastic.co/u/Artyom_Davydov)\ **Replies:** 5\ **Last updated:** [March 3, 2017, 8:00am UTC](https://discuss.elastic.co/t/problem-with-logstash-config/77275 "2017-03-03T08:00:57Z")
Hi all. We have trying to migrate our config (1.5) to newer (5.2) version of logstash . And we have some problem with multiline plugin. so here is an example of our config and log error. filter { if \[type\] == "osb\_ser… --- ## [Syslog output plugin log format is not proper](https://discuss.elastic.co/t/syslog-output-plugin-log-format-is-not-proper/77260)
**Author:** [@ronakg](https://discuss.elastic.co/u/ronakg)\ **Replies:** 1\ **Last updated:** [March 3, 2017, 7:45am UTC](https://discuss.elastic.co/t/syslog-output-plugin-log-format-is-not-proper/77260 "2017-03-03T07:45:52Z")
Hi, I'm using the syslog output plugin to send events out to a remote syslog server. Below is my configuration: output { syslog { host =\> "172.20.24.65" port =\> 514 appname =\> "%{\[kubernetes… --- ## [Consume from dB then update/delete?](https://discuss.elastic.co/t/consume-from-db-then-update-delete/77197)
**Author:** [@jsps](https://discuss.elastic.co/u/jsps)\ **Replies:** 1\ **Last updated:** [March 3, 2017, 4:57am UTC](https://discuss.elastic.co/t/consume-from-db-then-update-delete/77197 "2017-03-03T04:57:02Z")
Hello, we have logs being written to a database - I can consume these using the jdbc plugin, but wanted to update the corresponding DB table with some flag like "consumed" or even delete the rows that are consumed . D… --- ## [Geopoint generation](https://discuss.elastic.co/t/geopoint-generation/77231)
**Author:** [@sharon.c](https://discuss.elastic.co/u/sharon.c)\ **Replies:** 1\ **Last updated:** [March 3, 2017, 4:34am UTC](https://discuss.elastic.co/t/geopoint-generation/77231 "2017-03-03T04:34:26Z")
I am using Logstash 5.1.1, Elasticsearch 5.1 When I use the geoip feature, in logstash by feeding normal ip adresses, it shows the following warnings continuously. 2T23:04:13.930Z %{host} %{message}\], :response=\>{"in… --- ## [Logstash didn't split log as config say](https://discuss.elastic.co/t/logstash-didnt-split-log-as-config-say/77080)
**Author:** [@beenoisy](https://discuss.elastic.co/u/beenoisy)\ **Replies:** 2\ **Last updated:** [March 3, 2017, 3:25am UTC](https://discuss.elastic.co/t/logstash-didnt-split-log-as-config-say/77080 "2017-03-03T03:25:28Z")
I am using elkb to handle my access log. And some day, I found kibana missing a log. Then, grep filebeat log, And I can find the missing log: cat km-access-2017-03-01 |grep 17714317657896955-151 BTW 17714317657896… --- ## [Time/Date logic in logstash conf](https://discuss.elastic.co/t/time-date-logic-in-logstash-conf/76824)
**Author:** [@Keeper-of-the-Keys](https://discuss.elastic.co/u/Keeper-of-the-Keys)\ **Replies:** 2\ **Last updated:** [March 3, 2017, 2:42am UTC](https://discuss.elastic.co/t/time-date-logic-in-logstash-conf/76824 "2017-03-03T02:42:38Z")
Can I do any timestamp/date logic in the logstash configuration to make logstash have a cutoff date so it only sends events that happened after a certain date? Something like this: if \[@timestamp\] \> "2017-02-01T00:00:0… --- ## [Grok doesn't split fields properly](https://discuss.elastic.co/t/grok-doesnt-split-fields-properly/77047)
**Author:** [@kishorramanan](https://discuss.elastic.co/u/kishorramanan)\ **Replies:** 3\ **Last updated:** [March 3, 2017, 1:29am UTC](https://discuss.elastic.co/t/grok-doesnt-split-fields-properly/77047 "2017-03-03T01:29:51Z")
Log file: INFO 2017-02-24 14:25:12,612 ScriptEngines.java:191 - Loaded groovy ScriptEngine WARN 2017-02-24 14:25:12,814 DseSessionOpProcessor.java:295 - Exception processing a script on request \[RequestMessage{, mana… --- ## [Regex in conditionals](https://discuss.elastic.co/t/regex-in-conditionals/77090)
**Author:** [@timmy8ken](https://discuss.elastic.co/u/timmy8ken)\ **Replies:** 2\ **Last updated:** [March 3, 2017, 12:20am UTC](https://discuss.elastic.co/t/regex-in-conditionals/77090 "2017-03-03T00:20:45Z")
Hi All, I have log files that I need to parse that have multiple and varying lines, I have managed to write a number of grok queries that are successfully matching all lines in the log, all similar this: if !("p… --- ## [Tinyint field cannot be converted to integer](https://discuss.elastic.co/t/tinyint-field-cannot-be-converted-to-integer/77221)
**Author:** [@sharon.c](https://discuss.elastic.co/u/sharon.c)\ **Replies:** 1\ **Last updated:** [March 2, 2017, 11:04pm UTC](https://discuss.elastic.co/t/tinyint-field-cannot-be-converted-to-integer/77221 "2017-03-02T23:04:16Z")
I am using logstash 5.1.1 I ported data from mysql using mysql input plugin to elasticsearch. When I run the pipeline, for the tinyint fields, logstash cannot convert it to integer automatically. So I used mutation fi… --- ## [Logstash port data from Mysql, cannot handle tinyint type field defined from Mysql](https://discuss.elastic.co/t/logstash-port-data-from-mysql-cannot-handle-tinyint-type-field-defined-from-mysql/77227)
**Author:** [@sharon.c](https://discuss.elastic.co/u/sharon.c)\ **Replies:** 1\ **Last updated:** [March 2, 2017, 11:03pm UTC](https://discuss.elastic.co/t/logstash-port-data-from-mysql-cannot-handle-tinyint-type-field-defined-from-mysql/77227 "2017-03-02T23:03:11Z")
I defined the field as integer in elasticsearch type template, here is the screenshot of the definition in kibana I use logstash to port data from mysql to elasticsearch, when I turn on the pipeline, this response is … --- ## [Manage multiple Logstash instances from central place](https://discuss.elastic.co/t/manage-multiple-logstash-instances-from-central-place/76868)
**Author:** [@Jayanna\_Hallur](https://discuss.elastic.co/u/Jayanna_Hallur)\ **Replies:** 4\ **Last updated:** [March 2, 2017, 10:33pm UTC](https://discuss.elastic.co/t/manage-multiple-logstash-instances-from-central-place/76868 "2017-03-02T22:33:38Z")
I wanted to have 4-6 Logstash instances to parse/transforms lots of data. The configuration file for parsing on these files needs to be placed in all Logstash instance running on different systems. Later if I have to mo… --- ## [Logstash filter verifier - any working example?](https://discuss.elastic.co/t/logstash-filter-verifier-any-working-example/41880)
**Author:** [@terramexx](https://discuss.elastic.co/u/terramexx)\ **Replies:** 28\ **Last updated:** [March 2, 2017, 10:13pm UTC](https://discuss.elastic.co/t/logstash-filter-verifier-any-working-example/41880 "2017-03-02T22:13:54Z")
Continuing the discussion from Logstash test with rspec: The tool https://github.com/magnusbaeck/logstash-filter-verifier seems to be exactly what I was looking for: - easy install (in compare to rspec) - based on logs… --- ## [Elasticsearch Unreachable error in Logstash](https://discuss.elastic.co/t/elasticsearch-unreachable-error-in-logstash/75157)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 8\ **Last updated:** [March 2, 2017, 9:15pm UTC](https://discuss.elastic.co/t/elasticsearch-unreachable-error-in-logstash/75157 "2017-03-02T21:15:02Z")
Hi, I have just upgraded from ELK 5.0.0 to 5.2.0. I have installed and set X-Pack 5.2.0 for Elasticsearch, Logstash and Kibana, and enabled monitoring alone in Kibana and Elasticsearch. Kibana and Elasticsearch are wo… --- ## [The right way to specify environment?](https://discuss.elastic.co/t/the-right-way-to-specify-environment/77204)
**Author:** [@ybizeul](https://discuss.elastic.co/u/ybizeul)\ **Replies:** 3\ **Last updated:** [March 2, 2017, 8:07pm UTC](https://discuss.elastic.co/t/the-right-way-to-specify-environment/77204 "2017-03-02T20:07:29Z")
I'm trying to figure out how I am supposed to setup a consistent environment for logstash to startup correctly. # /usr/share/logstash/bin/system-install Successfully created system startup script for Logstash Looks … --- ## [Logstash HeapDumpOnOutOfMemoryError](https://discuss.elastic.co/t/logstash-heapdumponoutofmemoryerror/76161)
**Author:** [@higee](https://discuss.elastic.co/u/higee)\ **Replies:** 8\ **Last updated:** [March 2, 2017, 8:37am UTC](https://discuss.elastic.co/t/logstash-heapdumponoutofmemoryerror/76161 "2017-03-02T08:37:14Z")
Hi. I'm using Logstash to extract data from Database and send data to Elasticsearch. Everything works fine; data is well processed and sent to Elasticsearch without loss. The problem, however, is burden on the server. … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1024) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1026)