# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=968 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 969 --- ## [Logstash json parsing error to read logback json file](https://discuss.elastic.co/t/logstash-json-parsing-error-to-read-logback-json-file/88644)
**Author:** [@Juneyoung\_Oh](https://discuss.elastic.co/u/Juneyoung_Oh)\ **Replies:** 1\ **Last updated:** [June 9, 2017, 8:54am UTC](https://discuss.elastic.co/t/logstash-json-parsing-error-to-read-logback-json-file/88644 "2017-06-09T08:54:49Z")
Hi. I am trying to read and store tomcat logs into elasticsearch via logstash. The json file which made by logback looks like below. { "@timestamp" : "2017-06-08T09:33:48.579+09:00", "@version" : 1, "message" : "… --- ## [5.4.1 broke netflow codec!](https://discuss.elastic.co/t/5-4-1-broke-netflow-codec/88793)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 3\ **Last updated:** [June 9, 2017, 8:09am UTC](https://discuss.elastic.co/t/5-4-1-broke-netflow-codec/88793 "2017-06-09T08:09:50Z")
Hi, Big problem here. I upgraded from 5.4.0 to 5.4.1 and now I'm having big problems with netflow. On 5.4.0 I had two netflow v9 devices inputting data, this appeared to work fine though I did not reboot the server … --- ## [Logstash verify output availability](https://discuss.elastic.co/t/logstash-verify-output-availability/61396)
**Author:** [@Joao\_Paulo\_Karol\_Nun](https://discuss.elastic.co/u/Joao_Paulo_Karol_Nun)\ **Replies:** 9\ **Last updated:** [June 9, 2017, 6:28am UTC](https://discuss.elastic.co/t/logstash-verify-output-availability/61396 "2017-06-09T06:28:38Z")
Hi all, I wonder if there is a way to verify if a tool (elasticSearch etc) defined in the output filter is available before sending messages. The reason is because logstash is hanging when an output is not available and… --- ## [Dynamic input file name](https://discuss.elastic.co/t/dynamic-input-file-name/25951)
**Author:** [@Raman\_Saini](https://discuss.elastic.co/u/Raman_Saini)\ **Replies:** 8\ **Last updated:** [June 9, 2017, 5:30am UTC](https://discuss.elastic.co/t/dynamic-input-file-name/25951 "2017-06-09T05:30:24Z")
Hi, I have log file whose name changes as per date. How to set path in the input filter. i have given below path: "/www/redhat/ews-abc-uat/tomcat7/logs/access.date '+%F'.log" Its not working. Please suggest. Rega… --- ## [Help Filter JSON file](https://discuss.elastic.co/t/help-filter-json-file/88300)
**Author:** [@Rafael\_Pereira\_Silva](https://discuss.elastic.co/u/Rafael_Pereira_Silva)\ **Replies:** 3\ **Last updated:** [June 9, 2017, 5:22am UTC](https://discuss.elastic.co/t/help-filter-json-file/88300 "2017-06-09T05:22:14Z")
Hi, I have a record that is generated by an application, this log comes as JSON. But the log is coming broken. Follow Log: (2017-06-05 11:53:22) { "\_\_v": 0, "\_id": "Joq3PHwo8AHiUueiZ73VXpavSo84azUg30tydFVWpQTJcvMD4… --- ## [Logstash reading updated logs?](https://discuss.elastic.co/t/logstash-reading-updated-logs/88486)
**Author:** [@Aylwinns\_Tan](https://discuss.elastic.co/u/Aylwinns_Tan)\ **Replies:** 3\ **Last updated:** [June 9, 2017, 4:25am UTC](https://discuss.elastic.co/t/logstash-reading-updated-logs/88486 "2017-06-09T04:25:38Z")
Hi all, I would like to check if logstash is able to read updated logs instead of appended logs. For example, my program generates an XML log over time but the XML data is updated into the log file which means there … --- ## [How to track/monitor logstash errors](https://discuss.elastic.co/t/how-to-track-monitor-logstash-errors/88762)
**Author:** [@saisn](https://discuss.elastic.co/u/saisn)\ **Replies:** 1\ **Last updated:** [June 9, 2017, 3:44am UTC](https://discuss.elastic.co/t/how-to-track-monitor-logstash-errors/88762 "2017-06-09T03:44:36Z")
Hi, I've multiple logstash process setup and scheduled every 5 minutes. Here, I'd like to include a mechanism that capture any error or exception thrown by logstash process and eventually trigger an alert in the form o… --- ## [Logstash 5.4.0Unable to create Kafka producer from given configuration,Error registering plugin {:plugin=\>"#\ **Author:** [@anteoy](https://discuss.elastic.co/u/anteoy)\ **Replies:** 0\ **Last updated:** [June 9, 2017, 3:23am UTC](https://discuss.elastic.co/t/logstash-5-4-0unable-to-create-kafka-producer-from-given-configuration-error-registering-plugin-plugin-logstash/88783 "2017-06-09T03:23:57Z") my configuration like this: input { file { path =\> "/usr/share/logstash/agent/tmp/\*" start\_position =\> "beginning" } } filter { } output { kafka { bootstrap\_servers =\> "SASL\_SSL:/… --- ## [Logstash config to delete records in elastic?](https://discuss.elastic.co/t/logstash-config-to-delete-records-in-elastic/88366)
**Author:** [@west415](https://discuss.elastic.co/u/west415)\ **Replies:** 2\ **Last updated:** [June 8, 2017, 11:22pm UTC](https://discuss.elastic.co/t/logstash-config-to-delete-records-in-elastic/88366 "2017-06-08T23:22:59Z")
Hi, I'm using Logstash 5.1.1 and new to the ELK stack. I have a logstash config file setup that does an upsert of records from my Microsoft SQL Server database every 5 minutes into an index in Elastic. When a record… --- ## [Replace @timestamp with unix timestamp from MySQL](https://discuss.elastic.co/t/replace-timestamp-with-unix-timestamp-from-mysql/88428)
**Author:** [@Bdriss\_Sana](https://discuss.elastic.co/u/Bdriss_Sana)\ **Replies:** 3\ **Last updated:** [June 8, 2017, 9:23pm UTC](https://discuss.elastic.co/t/replace-timestamp-with-unix-timestamp-from-mysql/88428 "2017-06-08T21:23:44Z")
I'm trying to inject MySQL data to Elasticsearch using Logstash. This is my mysql table structure (id , datetime, value) where datetime consists if Unix Timestamp. ( example (131,1436298252,128) ) And here's the conf… --- ## [ELK Exception handling](https://discuss.elastic.co/t/elk-exception-handling/88743)
**Author:** [@sanjeevkt](https://discuss.elastic.co/u/sanjeevkt)\ **Replies:** 2\ **Last updated:** [June 8, 2017, 7:56pm UTC](https://discuss.elastic.co/t/elk-exception-handling/88743 "2017-06-08T19:56:23Z")
Hello, We have done ELK setup in out application and that works fine Issue: When ELK is down due to any reason then our main application is also not getting up, below is the exception Could not create SyslogWriter ja… --- ## [Creating an array field with mutate](https://discuss.elastic.co/t/creating-an-array-field-with-mutate/88635)
**Author:** [@nileshgattani](https://discuss.elastic.co/u/nileshgattani)\ **Replies:** 1\ **Last updated:** [June 8, 2017, 7:45pm UTC](https://discuss.elastic.co/t/creating-an-array-field-with-mutate/88635 "2017-06-08T19:45:16Z")
Hi, I am writing logstash filters for log parsing, and at one step I am using elasticsearch input plugin. The elastic search input plugin expects hosts as an array. Now, I am trying to access the hosts values as fi… --- ## [Grokparsefailure syslog which has no pid](https://discuss.elastic.co/t/grokparsefailure-syslog-which-has-no-pid/88353)
**Author:** [@Bhargav\_Narine](https://discuss.elastic.co/u/Bhargav_Narine)\ **Replies:** 2\ **Last updated:** [June 8, 2017, 7:37pm UTC](https://discuss.elastic.co/t/grokparsefailure-syslog-which-has-no-pid/88353 "2017-06-08T19:37:37Z")
Hi, I have written a plugin to resolve host names using DNS lookup. It was working fine and recently, dns is been blocked in lab. And, we have updated /etc/hosts for resolving it locally. Still, \_grokparse failure i… --- ## [Logstash HTTP polling performance](https://discuss.elastic.co/t/logstash-http-polling-performance/88368)
**Author:** [@juri](https://discuss.elastic.co/u/juri)\ **Replies:** 1\ **Last updated:** [June 8, 2017, 7:29pm UTC](https://discuss.elastic.co/t/logstash-http-polling-performance/88368 "2017-06-08T19:29:33Z")
Hello, In my use case I need to periodically query huge population of devices(~1M) via HTTP REST API and store status of each device in elasticsearch. I consider to use http\_poller but I worry about perfomance and s… --- ## [Multiline hybris stacktrace with different dates](https://discuss.elastic.co/t/multiline-hybris-stacktrace-with-different-dates/88730)
**Author:** [@Aleksandr\_Matveyev](https://discuss.elastic.co/u/Aleksandr_Matveyev)\ **Replies:** 0\ **Last updated:** [June 8, 2017, 3:17pm UTC](https://discuss.elastic.co/t/multiline-hybris-stacktrace-with-different-dates/88730 "2017-06-08T15:17:51Z")
Hello everybody, I have very bad issue. I need to parse a log such as: INFO | jvm 1 | main | 2015/04/11 17:25:58.569 | java.lang.NullPointerException INFO | jvm 1 | main | 2015/04/11 17:25:58.569 | at de… --- ## [Syntax for readable long lines in logstash config](https://discuss.elastic.co/t/syntax-for-readable-long-lines-in-logstash-config/88754)
**Author:** [@selivan](https://discuss.elastic.co/u/selivan)\ **Replies:** 2\ **Last updated:** [June 8, 2017, 7:01pm UTC](https://discuss.elastic.co/t/syntax-for-readable-long-lines-in-logstash-config/88754 "2017-06-08T19:01:55Z")
I am using a very long grok templates to parse some complex logs like mysql slow log. It looks ugly in config: very long string, full of characters like (?\%{ZZ})? - hard to read and hard to write. Does logstash config… --- ## [Logstash - Charset 1252 - Problems with Conversion](https://discuss.elastic.co/t/logstash-charset-1252-problems-with-conversion/87729)
**Author:** [@lgarcia](https://discuss.elastic.co/u/lgarcia)\ **Replies:** 5\ **Last updated:** [June 8, 2017, 5:08pm UTC](https://discuss.elastic.co/t/logstash-charset-1252-problems-with-conversion/87729 "2017-06-08T17:08:56Z")
Hi everyone I have been struggling with Logstash to be able to apply the windows charset CP1252 in a UDP input listener. I'm running on a windows server 2012 with Logstash 5.2.2. The charset setup seems does not make… --- ## [CSV with different number of columns](https://discuss.elastic.co/t/csv-with-different-number-of-columns/88744)
**Author:** [@averdoy](https://discuss.elastic.co/u/averdoy)\ **Replies:** 0\ **Last updated:** [June 8, 2017, 4:52pm UTC](https://discuss.elastic.co/t/csv-with-different-number-of-columns/88744 "2017-06-08T16:52:00Z")
We generate some log files in csv comma separated format that we process with Logstash to index into Elasticsearch. Everything is working fine, but last week we started to generate those csv files adding a new column, … --- ## [Multiline codec query](https://discuss.elastic.co/t/multiline-codec-query/88733)
**Author:** [@Raghu19](https://discuss.elastic.co/u/Raghu19)\ **Replies:** 1\ **Last updated:** [June 8, 2017, 4:37pm UTC](https://discuss.elastic.co/t/multiline-codec-query/88733 "2017-06-08T16:37:20Z")
dear all, Greetings! I am trying to use logstash to parse and load the application logs and request expert advice her with multi line log msgs using logstash 5.1.2 The log file starts with the below lines FILE\_TYPE: En… --- ## [Using metric with dynamic field names](https://discuss.elastic.co/t/using-metric-with-dynamic-field-names/88441)
**Author:** [@kbop543](https://discuss.elastic.co/u/kbop543)\ **Replies:** 2\ **Last updated:** [June 8, 2017, 3:14pm UTC](https://discuss.elastic.co/t/using-metric-with-dynamic-field-names/88441 "2017-06-08T15:14:16Z")
Hey, So I have been testing how to use metrics to tell me the count of how many times a logfile logged errors. I use indexes to distinguish between logfiles by grabbing the filename from the path in a field called fil… --- ## [Logstash unhandled exception issue](https://discuss.elastic.co/t/logstash-unhandled-exception-issue/88725)
**Author:** [@can.zhang](https://discuss.elastic.co/u/can.zhang)\ **Replies:** 0\ **Last updated:** [June 8, 2017, 2:45pm UTC](https://discuss.elastic.co/t/logstash-unhandled-exception-issue/88725 "2017-06-08T14:45:25Z")
Does anyone have experience handle this issue? The logstash error log like this. {:timestamp=\>"2017-06-08T14:09:18.074000+0000", :message=\>"Beats input: unhandled exception", :exception=\>#, :backtrace=\>\["org/jruby/ext/… --- ## [Logstash-Filter-CSV plugin clarification (skip header and define column )](https://discuss.elastic.co/t/logstash-filter-csv-plugin-clarification-skip-header-and-define-column/88720)
**Author:** [@Ajay1](https://discuss.elastic.co/u/Ajay1)\ **Replies:** 0\ **Last updated:** [June 8, 2017, 2:06pm UTC](https://discuss.elastic.co/t/logstash-filter-csv-plugin-clarification-skip-header-and-define-column/88720 "2017-06-08T14:06:44Z")
Hi All, I recently upgraded Logstash from 5.0 to 5.3 and using Filter-csv plugin for some of the reports . My configuration goes fine without any changes but in Kibana i could not get the data in column specific (exp… --- ## [Cannot manage nested fields](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293)
**Author:** [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\ **Replies:** 8\ **Last updated:** [June 8, 2017, 2:02pm UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293 "2017-06-08T14:02:47Z")
Hi, could you please help to understand why removing of nested fields from 'grok' and 'mutate' is not working in my case, or how to remove them properly via filter settings, or any well know best practices regarding thi… --- ## [Logstash output to webhdfs](https://discuss.elastic.co/t/logstash-output-to-webhdfs/88715)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 0\ **Last updated:** [June 8, 2017, 12:20pm UTC](https://discuss.elastic.co/t/logstash-output-to-webhdfs/88715 "2017-06-08T12:20:06Z")
Need to split the logstash output to hadoop, per minute. My output looks like that: webhdfs { host =\> "127.0.0.1" # (required) port =\> 50070 # (optional, default: 50070) … --- ## [Nested json imposible to parse](https://discuss.elastic.co/t/nested-json-imposible-to-parse/88669)
**Author:** [@fernando\_martinez](https://discuss.elastic.co/u/fernando_martinez)\ **Replies:** 6\ **Last updated:** [June 8, 2017, 12:08pm UTC](https://discuss.elastic.co/t/nested-json-imposible-to-parse/88669 "2017-06-08T12:08:08Z")
Hi, I have a nested json in a field like this: { .... "host" =\> "repo1.server.io", "source" =\> "/mnt/centralizedlogs/logstash/netsec/48851f14d3e9/2017-06-08.txt", "message" =\> "{\\"@timestamp\\"=\>\\"2017-06-08T07:06:13.… --- ## [Logstash file for parsing JSON data](https://discuss.elastic.co/t/logstash-file-for-parsing-json-data/88570)
**Author:** [@patil-akshay](https://discuss.elastic.co/u/patil-akshay)\ **Replies:** 5\ **Last updated:** [June 8, 2017, 11:42am UTC](https://discuss.elastic.co/t/logstash-file-for-parsing-json-data/88570 "2017-06-08T11:42:25Z")
I want to parse the below-given JSON data through logstash and load it to the KIbana Data {"PhoneNumber":{"0":27817768541},"TimeStamp":{"0":1471433451000},"ncalls":{"0":20}} can anyone help me with the logstash co… --- ## [Logstash not show any output](https://discuss.elastic.co/t/logstash-not-show-any-output/88445)
**Author:** [@baha1](https://discuss.elastic.co/u/baha1)\ **Replies:** 15\ **Last updated:** [June 8, 2017, 10:58am UTC](https://discuss.elastic.co/t/logstash-not-show-any-output/88445 "2017-06-08T10:58:12Z")
Hi there, i have LS and ES are runing,but there is nothing printed in ES or LS. this is pipeline config. input { file { path =\> "access\_log.txt" } } filter { grok { match =\> { "message" =\> "%{COMBINEDAPACHELOG}" } } … --- ## [Logstash jvm stats api heap used percent](https://discuss.elastic.co/t/logstash-jvm-stats-api-heap-used-percent/88407)
**Author:** [@Exocomp](https://discuss.elastic.co/u/Exocomp)\ **Replies:** 1\ **Last updated:** [June 8, 2017, 10:24am UTC](https://discuss.elastic.co/t/logstash-jvm-stats-api-heap-used-percent/88407 "2017-06-08T10:24:26Z")
Hi, I have question regarding the jvm info returned with the logstash node stats API (I'm on version 5.3.1). API =\> "http://127.0.0.1:9600/\_node/stats/jvm" Here is what it returns: heap\_used\_percent =\> 51 % heap\_us… --- ## [Create and split some fields](https://discuss.elastic.co/t/create-and-split-some-fields/88524)
**Author:** [@Jean-Marc\_Glevarec](https://discuss.elastic.co/u/Jean-Marc_Glevarec)\ **Replies:** 7\ **Last updated:** [June 8, 2017, 7:23am UTC](https://discuss.elastic.co/t/create-and-split-some-fields/88524 "2017-06-08T07:23:45Z")
Hello, I have some lines in an EAD.stats.log file like that : DocType CNI: # DateEmission = 100% # DateNaissance = 100% # DateValidite = 100% # DepartementDelivrance = 100% # LieuNaissance = 10% # MRZ = 100% # MRZ2 =… --- ## [Rule identification in messages](https://discuss.elastic.co/t/rule-identification-in-messages/88281)
**Author:** [@isaacpm](https://discuss.elastic.co/u/isaacpm)\ **Replies:** 2\ **Last updated:** [June 8, 2017, 7:22am UTC](https://discuss.elastic.co/t/rule-identification-in-messages/88281 "2017-06-08T07:22:41Z")
Hi, We started using logstash not long ago and the number of rules is growing quite quickly. Which makes it difficult to know which rule is applied to each message in elasticsearch. Is there a standard or easy way to… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=967) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=969)