# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=494 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 495 --- ## [Filebeat passing over logstash](https://discuss.elastic.co/t/filebeat-passing-over-logstash/194882)
**Author:** [@Felipe\_Cabral\_Jeroni](https://discuss.elastic.co/u/Felipe_Cabral_Jeroni)\ **Replies:** 2\ **Last updated:** [August 13, 2019, 7:46am UTC](https://discuss.elastic.co/t/filebeat-passing-over-logstash/194882 "2019-08-13T07:46:29Z")
Hi guys, I'm trying to figure out why my filebeat is not sending information to logstach. Instead, it's sending directly to elastic. When I go to my kibana there is only the filebeat index patter. When I access the ela… --- ## [JSON data split up (\_jsonparseerror) in logstash-7.2.0-1. Works fine in logstash-7.1.1-1](https://discuss.elastic.co/t/json-data-split-up-jsonparseerror-in-logstash-7-2-0-1-works-fine-in-logstash-7-1-1-1/190624)
**Author:** [@sjaganna](https://discuss.elastic.co/u/sjaganna)\ **Replies:** 5\ **Last updated:** [August 13, 2019, 7:22am UTC](https://discuss.elastic.co/t/json-data-split-up-jsonparseerror-in-logstash-7-2-0-1-works-fine-in-logstash-7-1-1-1/190624 "2019-08-13T07:22:33Z")
Hello! I've been in the midst of working with the ELK stack for the first time. I'm currently sending data from Metricbeat -\> Logstash -\> an external script (output sent over TCP) -\> Logstash -\> Elasticsearch. However, … --- ## [Logstash installation on AWS EC2 instance](https://discuss.elastic.co/t/logstash-installation-on-aws-ec2-instance/194961)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 1\ **Last updated:** [August 13, 2019, 7:13am UTC](https://discuss.elastic.co/t/logstash-installation-on-aws-ec2-instance/194961 "2019-08-13T07:13:15Z")
Would like to know which type of AWS EC2 instance is suitable for installing the logstash. --- ## [\_dateparseerror](https://discuss.elastic.co/t/dateparseerror/194889)
**Author:** [@Sajal](https://discuss.elastic.co/u/Sajal)\ **Replies:** 5\ **Last updated:** [August 13, 2019, 3:06am UTC](https://discuss.elastic.co/t/dateparseerror/194889 "2019-08-13T03:06:48Z")
How to parse 2019-08-12T03:09:13.142635Z ? --- ## [Unable to perform offline upgrade of logstash plugins 2](https://discuss.elastic.co/t/unable-to-perform-offline-upgrade-of-logstash-plugins-2/194938)
**Author:** [@johncollaros](https://discuss.elastic.co/u/johncollaros)\ **Replies:** 0\ **Last updated:** [August 13, 2019, 12:43am UTC](https://discuss.elastic.co/t/unable-to-perform-offline-upgrade-of-logstash-plugins-2/194938 "2019-08-13T00:43:42Z")
This topic was closed, but the issue still exists. I have only been able to get offline-install to work once, when a minor upgrade of a logstash-input-beats was done. Where it is a big jump and lots of dependencies i… --- ## [Import CSV file to Logstash on windows10](https://discuss.elastic.co/t/import-csv-file-to-logstash-on-windows10/194770)
**Author:** [@aahtysh2525](https://discuss.elastic.co/u/aahtysh2525)\ **Replies:** 2\ **Last updated:** [August 12, 2019, 10:29pm UTC](https://discuss.elastic.co/t/import-csv-file-to-logstash-on-windows10/194770 "2019-08-12T22:29:27Z")
Hi, I am trying to import a CSV file to elasticsearch using logstash, but with no success. Windows 10 Logstash 7.3.0/Elasticsearch 7.3.0/Kibana 7.3.0 // Config File is as below input { file { path =\> \["E:\\Softwa… --- ## [Issue with KV Filter when value ends with escape character](https://discuss.elastic.co/t/issue-with-kv-filter-when-value-ends-with-escape-character/190618)
**Author:** [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Replies:** 4\ **Last updated:** [August 12, 2019, 9:39pm UTC](https://discuss.elastic.co/t/issue-with-kv-filter-when-value-ends-with-escape-character/190618 "2019-08-12T21:39:18Z")
I am having an issue parsing some logs, and I would appreciate if someone could assist. I have logs coming in that are primarily handled with a KV Filter. This works fine on about 99% of the messages, but one message ty… --- ## [Logstash enriching data with lookup](https://discuss.elastic.co/t/logstash-enriching-data-with-lookup/194909)
**Author:** [@philippkahr](https://discuss.elastic.co/u/philippkahr)\ **Replies:** 2\ **Last updated:** [August 12, 2019, 8:28pm UTC](https://discuss.elastic.co/t/logstash-enriching-data-with-lookup/194909 "2019-08-12T20:28:31Z")
Hi guys, I currently struggeling in understanding how this can be done within the elastic stack properly. I take netflow data as an input source in netflow. Specifically, I use elastiflow https://github.com/robcowart/el… --- ## ["this field is present in your elasticsearch mapping" on \_source field](https://discuss.elastic.co/t/this-field-is-present-in-your-elasticsearch-mapping-on-source-field/190308)
**Author:** [@hueyg](https://discuss.elastic.co/u/hueyg)\ **Replies:** 5\ **Last updated:** [August 12, 2019, 8:12pm UTC](https://discuss.elastic.co/t/this-field-is-present-in-your-elasticsearch-mapping-on-source-field/190308 "2019-08-12T20:12:46Z")
Upgraded my ELK stack to 6.8 finally. Beats are on 6.4 still. Zero issues all last week and all this week. Today I noticed one of my filters for one of my beats was experiencing some parsing errors on certain log entr… --- ## [How to Handle Exceptions?](https://discuss.elastic.co/t/how-to-handle-exceptions/194900)
**Author:** [@cd-agutierrez](https://discuss.elastic.co/u/cd-agutierrez)\ **Replies:** 4\ **Last updated:** [August 12, 2019, 7:58pm UTC](https://discuss.elastic.co/t/how-to-handle-exceptions/194900 "2019-08-12T19:58:06Z")
It seems that if logstash encounters an exception in the elasticsearch output pipeline it will completely halt processing of any further entries. \[2019-08-08T22:14:44,075\]\[WARN \]\[io.netty.channel.DefaultChannelPipeline\]… --- ## [Pipeline Output Filter not accepting existing fields from an event as parameter](https://discuss.elastic.co/t/pipeline-output-filter-not-accepting-existing-fields-from-an-event-as-parameter/194899)
**Author:** [@perryparktung](https://discuss.elastic.co/u/perryparktung)\ **Replies:** 2\ **Last updated:** [August 12, 2019, 7:14pm UTC](https://discuss.elastic.co/t/pipeline-output-filter-not-accepting-existing-fields-from-an-event-as-parameter/194899 "2019-08-12T19:14:42Z")
Hi, Im trying out pipeline-to-pipeline connections (https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html), and apparently the output filter does not recognize existing fields of an event as a param… --- ## [DNS lookup how to drop log filed if not resolved?](https://discuss.elastic.co/t/dns-lookup-how-to-drop-log-filed-if-not-resolved/194878)
**Author:** [@iukea](https://discuss.elastic.co/u/iukea)\ **Replies:** 6\ **Last updated:** [August 12, 2019, 7:02pm UTC](https://discuss.elastic.co/t/dns-lookup-how-to-drop-log-filed-if-not-resolved/194878 "2019-08-12T19:02:02Z")
Hello, I am running logstash 6.8 I am taking all Src\_Ip addresses and doing a reverse DNS lookup through the DNS plugin in logstash. Logstash YML if \[src\_ip\] { mutate { add\_field =\> {"DNS… --- ## [Logstash Persistent Variables](https://discuss.elastic.co/t/logstash-persistent-variables/194853)
**Author:** [@sho1](https://discuss.elastic.co/u/sho1)\ **Replies:** 9\ **Last updated:** [August 12, 2019, 6:07pm UTC](https://discuss.elastic.co/t/logstash-persistent-variables/194853 "2019-08-12T18:07:16Z")
I wanted to create a persistent boolean that I can change whenever a certain message appears. Based on this boolean I would change my output{ elastic search{ index file reference}}. E.g. log GET log dog Log PUT log … --- ## [Logstash invalid byte sequence in UTF-8 jdbc input](https://discuss.elastic.co/t/logstash-invalid-byte-sequence-in-utf-8-jdbc-input/194876)
**Author:** [@nick36](https://discuss.elastic.co/u/nick36)\ **Replies:** 2\ **Last updated:** [August 12, 2019, 5:30pm UTC](https://discuss.elastic.co/t/logstash-invalid-byte-sequence-in-utf-8-jdbc-input/194876 "2019-08-12T17:30:50Z")
Hi, I have a problem with an error invalid byte sequence in UTF-8. Logstash 7.3.0. conf below: input{ jdbc{ jdbc\_driver\_library =\> "C:\\sqljdbc42.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriv… --- ## [Firehose input with LogStash? Some confusion](https://discuss.elastic.co/t/firehose-input-with-logstash-some-confusion/194891)
**Author:** [@Neje](https://discuss.elastic.co/u/Neje)\ **Replies:** 0\ **Last updated:** [August 12, 2019, 4:53pm UTC](https://discuss.elastic.co/t/firehose-input-with-logstash-some-confusion/194891 "2019-08-12T16:53:46Z")
All, I'm a little confused and looking for clarification about the Logstash Kinesis input plug-in. As I read it, and from some cursory discussion here and other places on the web, it appears that it is possible to send… --- ## [Logstash - undefined method 'to\_msgpack' for 0.2e2:BigDecimal](https://discuss.elastic.co/t/logstash-undefined-method-to-msgpack-for-0-2e2-bigdecimal/194887)
**Author:** [@dhammond22222](https://discuss.elastic.co/u/dhammond22222)\ **Replies:** 0\ **Last updated:** [August 12, 2019, 4:15pm UTC](https://discuss.elastic.co/t/logstash-undefined-method-to-msgpack-for-0-2e2-bigdecimal/194887 "2019-08-12T16:15:53Z")
Hey all, Recently have been doing some JSON parsing of messages and it appears that every once in awhile, we come across a message that just breaks logstash and crashes it with a fatal error. I am using the s3-sns-sqs … --- ## [TypeError: can't convert Hash into String](https://discuss.elastic.co/t/typeerror-cant-convert-hash-into-string/194869)
**Author:** [@YanivT](https://discuss.elastic.co/u/YanivT)\ **Replies:** 5\ **Last updated:** [August 12, 2019, 3:17pm UTC](https://discuss.elastic.co/t/typeerror-cant-convert-hash-into-string/194869 "2019-08-12T15:17:40Z")
\[2019-08-12T10:57:09,072\]\[FATAL\]\[logstash.runner \] An unexpected error occurred! {:error=\>#\, :backtrace=\>\["org/jruby/RubyString.java:4462:in include?'", "(eval):6774:ini… --- ## [Extend file output plugin](https://discuss.elastic.co/t/extend-file-output-plugin/194861)
**Author:** [@Nitin\_Gupta3](https://discuss.elastic.co/u/Nitin_Gupta3)\ **Replies:** 0\ **Last updated:** [August 12, 2019, 2:24pm UTC](https://discuss.elastic.co/t/extend-file-output-plugin/194861 "2019-08-12T14:24:00Z")
Hi, I want to customise file output plugin such that it sorts the chunks according to a field before writing them. I'm looking for extending the existing file plugin rather than duplicating its code, but facing issues i… --- ## [Filebeat injest does not show up in logstash logs](https://discuss.elastic.co/t/filebeat-injest-does-not-show-up-in-logstash-logs/194849)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 2\ **Last updated:** [August 12, 2019, 1:33pm UTC](https://discuss.elastic.co/t/filebeat-injest-does-not-show-up-in-logstash-logs/194849 "2019-08-12T13:33:07Z")
Hello, I am injesting some java logs via filebeat. I see the logs being sent to logstash however on the stdout filter I don't see anything being processed by the logstash. But logstash does not throw any errors either. … --- ## [While overriding the @timestamp with log date , i am getting the time variation](https://discuss.elastic.co/t/while-overriding-the-timestamp-with-log-date-i-am-getting-the-time-variation/194753)
**Author:** [@satyam2593](https://discuss.elastic.co/u/satyam2593)\ **Replies:** 4\ **Last updated:** [August 12, 2019, 1:26pm UTC](https://discuss.elastic.co/t/while-overriding-the-timestamp-with-log-date-i-am-getting-the-time-variation/194753 "2019-08-12T13:26:38Z")
Why i am getting the time variation while overriding the @timestamp with logdate My input: INFO | jvm 1 | 2019/05/06 23:39:05.254 | FileName:xxxx\_20190506 File Size:4096 File Collection time:2019-05-06 23:39:05.181 … --- ## [Kafka output plugin is not picking up value from @metadata field](https://discuss.elastic.co/t/kafka-output-plugin-is-not-picking-up-value-from-metadata-field/194825)
**Author:** [@amrutadale](https://discuss.elastic.co/u/amrutadale)\ **Replies:** 1\ **Last updated:** [August 12, 2019, 12:37pm UTC](https://discuss.elastic.co/t/kafka-output-plugin-is-not-picking-up-value-from-metadata-field/194825 "2019-08-12T12:37:07Z")
Hello Team, I have http endpoint as input. Based on the value received in header tag for application , I need to generate the jass configuration at run time for kafka output topic. input { http { host =\> "127.0.0.1" … --- ## [Parsing Json object array](https://discuss.elastic.co/t/parsing-json-object-array/190954)
**Author:** [@Shawcs](https://discuss.elastic.co/u/Shawcs)\ **Replies:** 9\ **Last updated:** [August 12, 2019, 12:31pm UTC](https://discuss.elastic.co/t/parsing-json-object-array/190954 "2019-08-12T12:31:01Z")
Hi, I have trouble with a dynamic Json file. My json have an array of "circuitPath". This circuit path is a nested array inside my main json file that can be N element long. I would like to parse it dynamicaly like s… --- ## [How to change a number](https://discuss.elastic.co/t/how-to-change-a-number/194832)
**Author:** [@Arnodl](https://discuss.elastic.co/u/Arnodl)\ **Replies:** 1\ **Last updated:** [August 12, 2019, 11:37am UTC](https://discuss.elastic.co/t/how-to-change-a-number/194832 "2019-08-12T11:37:49Z")
hi, I'm using grok to pull out the user\_id field grok { match =\> { "authDN" =\> "uid=%{GREEDYDATA:user\_id}," } } My field user\_id=0212345678 I always need to delete the first 0 and insert 421. HOW? 0212345678 -\> 42121… --- ## [Logstash doesn't load all files in the folder](https://discuss.elastic.co/t/logstash-doesnt-load-all-files-in-the-folder/194821)
**Author:** [@daniele.saccon](https://discuss.elastic.co/u/daniele.saccon)\ **Replies:** 0\ **Last updated:** [August 12, 2019, 9:42am UTC](https://discuss.elastic.co/t/logstash-doesnt-load-all-files-in-the-folder/194821 "2019-08-12T09:42:37Z")
Hi, I'm using Logstash 6.8.1 with mode =\> "read". Logstash doesn't load all the files that respect the path and the regex specified in input and the unread files don't appear in the file specified in file\_completed\_log… --- ## [Failed to flush outgoing items, exception WebHDFS::IOError](https://discuss.elastic.co/t/failed-to-flush-outgoing-items-exception-webhdfs-ioerror/194777)
**Author:** [@pro\_sy](https://discuss.elastic.co/u/pro_sy)\ **Replies:** 0\ **Last updated:** [August 12, 2019, 3:06am UTC](https://discuss.elastic.co/t/failed-to-flush-outgoing-items-exception-webhdfs-ioerror/194777 "2019-08-12T03:06:50Z")
I've been trying to send my logs from server by beat and to hdfs server in compressed format. And this gave me some warnings and logs seems not to be sent over to hdfs unlike it used to when I didn't use compress options… --- ## [Tags not applied by json filter when source sets HTTP header Content-Type: application/json](https://discuss.elastic.co/t/tags-not-applied-by-json-filter-when-source-sets-http-header-content-type-application-json/194754)
**Author:** [@nnet](https://discuss.elastic.co/u/nnet)\ **Replies:** 3\ **Last updated:** [August 11, 2019, 10:24pm UTC](https://discuss.elastic.co/t/tags-not-applied-by-json-filter-when-source-sets-http-header-content-type-application-json/194754 "2019-08-11T22:24:38Z")
Hi, I set up an http listener, and json filter and discovered tag application inconsistencies. ELK components 5.6.16. Input: input { http { port =\> "5048" tags =\> \[ "CURL" \] type =\> "curl" } } Filter: filt… --- ## [Another Way to Stop/Restart Logstash inside 7.3.0 Docker Container?](https://discuss.elastic.co/t/another-way-to-stop-restart-logstash-inside-7-3-0-docker-container/194100)
**Author:** [@redapplesonly](https://discuss.elastic.co/u/redapplesonly)\ **Replies:** 1\ **Last updated:** [August 11, 2019, 8:51pm UTC](https://discuss.elastic.co/t/another-way-to-stop-restart-logstash-inside-7-3-0-docker-container/194100 "2019-08-11T20:51:43Z")
Hi Elasticfolk, I’m new to Logstash / Elasticsearch / Kibana, slowly building my first pipeline. I’ve pulled and spun up the Docker 7.3.0 versions of L/E/K. I’m concentrating on Logstash at the moment. So right off t… --- ## [Installed logstash, cannot find logstash-plugin](https://discuss.elastic.co/t/installed-logstash-cannot-find-logstash-plugin/194722)
**Author:** [@Jobe](https://discuss.elastic.co/u/Jobe)\ **Replies:** 2\ **Last updated:** [August 11, 2019, 3:45pm UTC](https://discuss.elastic.co/t/installed-logstash-cannot-find-logstash-plugin/194722 "2019-08-11T15:45:21Z")
I'm not sure if this is a version issue, I found a "plugin" executable in the bin directory that seems to be the same but I'm concerned that all the docs reference logstash-plugin and I cannot seem to find it in a basic … --- ## [How to parse this json](https://discuss.elastic.co/t/how-to-parse-this-json/194682)
**Author:** [@Sketchy](https://discuss.elastic.co/u/Sketchy)\ **Replies:** 6\ **Last updated:** [August 11, 2019, 2:50pm UTC](https://discuss.elastic.co/t/how-to-parse-this-json/194682 "2019-08-11T14:50:24Z")
I am new to logstash and have been trying to parse the below json with no luck. { "stat": "user", "Ref": "USER:\[5000,John Smith,3A37332D2659554F9CCE0CD754185269\]", "statistics": \[ "0", "", "", "0", 0… --- ## [How to create grok filter based on targeted word in message?](https://discuss.elastic.co/t/how-to-create-grok-filter-based-on-targeted-word-in-message/194686)
**Author:** [@Bob94](https://discuss.elastic.co/u/Bob94)\ **Replies:** 6\ **Last updated:** [August 11, 2019, 1:00pm UTC](https://discuss.elastic.co/t/how-to-create-grok-filter-based-on-targeted-word-in-message/194686 "2019-08-11T13:00:37Z")
Hi guys please verify whether this is the proper way to use grok ? If "i am happy: true" in \[message\] { grok { match =\> "message" , "%{GREEDYDATA:myfield"} } My intension is to create a new field based on spesific st… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=493) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=495)