# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1130 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1131 --- ## [Logstash Grok Parsing Issue](https://discuss.elastic.co/t/logstash-grok-parsing-issue/55010)
**Author:** [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\ **Replies:** 8\ **Last updated:** [July 8, 2016, 6:31am UTC](https://discuss.elastic.co/t/logstash-grok-parsing-issue/55010 "2016-07-08T06:31:03Z")
I am using Logstash to read some log file. Here are some data sources records \<2016-07-07 00:31:01\> Start \<2016-07-07 00:31:59\> Warning - Export\_Sysem 6 (1) =\> No records to be exported \<2016-07-07 00:32:22\> Export2… --- ## [How to stop Logstash running in background](https://discuss.elastic.co/t/how-to-stop-logstash-running-in-background/55018)
**Author:** [@gayathri](https://discuss.elastic.co/u/gayathri)\ **Replies:** 1\ **Last updated:** [July 8, 2016, 6:23am UTC](https://discuss.elastic.co/t/how-to-stop-logstash-running-in-background/55018 "2016-07-08T06:23:17Z")
Hi Team, I tried "/opt/logstash/bin/logstash agent -f /opt/logstash/conf.d/logstash.conf & " command for starting the logstash in background...but i want to stop this now... Pid is creating again and again even after k… --- ## [Getting duplicate Log entries in Redis](https://discuss.elastic.co/t/getting-duplicate-log-entries-in-redis/54650)
**Author:** [@gayathri](https://discuss.elastic.co/u/gayathri)\ **Replies:** 6\ **Last updated:** [July 8, 2016, 6:15am UTC](https://discuss.elastic.co/t/getting-duplicate-log-entries-in-redis/54650 "2016-07-08T06:15:53Z")
Hi Team, When I am trying to push the real time transaction logs to Redis server, redis recieves two copies of each log in production..... I checked the conf.d folder, there was two config files... But when I tried… --- ## [Check that a message is in JSON to avoid \_jsonparsefailure](https://discuss.elastic.co/t/check-that-a-message-is-in-json-to-avoid--jsonparsefailure/54959)
**Author:** [@norandom](https://discuss.elastic.co/u/norandom)\ **Replies:** 1\ **Last updated:** [July 8, 2016, 5:44am UTC](https://discuss.elastic.co/t/check-that-a-message-is-in-json-to-avoid--jsonparsefailure/54959 "2016-07-08T05:44:30Z")
Hi, I have a logfile, which sadly is mixed. There are JSON lines, and there are free text lines. The good news is, that there is always a line break and JSON parsing works. Currently I do: if "\_jsonparsefailure… --- ## [Error while transferring data in to Elastic Search Any one who can provide solution?](https://discuss.elastic.co/t/error-while-transferring-data-in-to-elastic-search-any-one-who-can-provide-solution/54986)
**Author:** [@punit\_sachdev](https://discuss.elastic.co/u/punit_sachdev)\ **Replies:** 2\ **Last updated:** [July 8, 2016, 5:36am UTC](https://discuss.elastic.co/t/error-while-transferring-data-in-to-elastic-search-any-one-who-can-provide-solution/54986 "2016-07-08T05:36:12Z")
When trying to load CSV file this error is coming in Logstash E:\\ES\\logstash-2.3.3\\bin\>logstash -f E:\\logtashtutorial.conf io/console not supported; tty will not be manipulated Settings: Default pipeline workers: 4 ←\[3… --- ## [How to detect when the input data is finished](https://discuss.elastic.co/t/how-to-detect-when-the-input-data-is-finished/54945)
**Author:** [@Ben\_Ryan](https://discuss.elastic.co/u/Ben_Ryan)\ **Replies:** 2\ **Last updated:** [July 8, 2016, 5:26am UTC](https://discuss.elastic.co/t/how-to-detect-when-the-input-data-is-finished/54945 "2016-07-08T05:26:40Z")
I am using Logstash from the cmd line with file input and using cat to push new data to the input file. I need to be able to detect when the input is finished so that I can kill the Logstash instance. --- ## [\_grokparsefailure when not using grok](https://discuss.elastic.co/t/-grokparsefailure-when-not-using-grok/54883)
**Author:** [@memelet](https://discuss.elastic.co/u/memelet)\ **Replies:** 3\ **Last updated:** [July 8, 2016, 5:21am UTC](https://discuss.elastic.co/t/-grokparsefailure-when-not-using-grok/54883 "2016-07-08T05:21:59Z")
I'm using file to ingest consul-template logs input { file { type =\> "consul-template" path =\> \["/var/log/consul-template.log"\] add\_field =\> { "name" =\> "consul-template" } } } For every … --- ## [Selectively filtering messages log](https://discuss.elastic.co/t/selectively-filtering-messages-log/55012)
**Author:** [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\ **Replies:** 1\ **Last updated:** [July 8, 2016, 4:52am UTC](https://discuss.elastic.co/t/selectively-filtering-messages-log/55012 "2016-07-08T04:52:46Z")
I have come across with some issues in logging some data. This log file contains various data. 2016-07-07 13:30:02 \[Main\] \*\*\* Program start \*\*\* 2016-07-07 13:30:02 \[UnzipFile\] Before file collection 2016-07-07 13:3… --- ## [How to parse log file with different types of messages](https://discuss.elastic.co/t/how-to-parse-log-file-with-different-types-of-messages/54910)
**Author:** [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\ **Replies:** 1\ **Last updated:** [July 7, 2016, 8:16am UTC](https://discuss.elastic.co/t/how-to-parse-log-file-with-different-types-of-messages/54910 "2016-07-07T08:16:35Z")
Hi everyone, I have a log file which contains complicated message types. Here is an example: 2016-07-07 13:30:02 \[UnzipFile\] Before file collection 2016-07-07 13:30:02 \[GetZipCol\] Start get sorted zip file collection… --- ## [File plugin: too many files error](https://discuss.elastic.co/t/file-plugin-too-many-files-error/54893)
**Author:** [@changxy](https://discuss.elastic.co/u/changxy)\ **Replies:** 2\ **Last updated:** [July 8, 2016, 1:14am UTC](https://discuss.elastic.co/t/file-plugin-too-many-files-error/54893 "2016-07-08T01:14:52Z")
IdentityMapCodec has reached 80% capacity {:current\_size=\>16427, :upper\_limit=\>20000, :level=\>:warn} how to fix it? $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited sc… --- ## [Modifying the timestamp](https://discuss.elastic.co/t/modifying-the-timestamp/55002)
**Author:** [@Mormaii](https://discuss.elastic.co/u/Mormaii)\ **Replies:** 5\ **Last updated:** [July 8, 2016, 1:01am UTC](https://discuss.elastic.co/t/modifying-the-timestamp/55002 "2016-07-08T01:01:30Z")
I've got a filebeat daemon outputting logs to my redis server. My logstash is getting the input from redis and the timestamps are looking like this: "2016-07-07T23:49:19.882Z". I'm trying to modify it with this logstash … --- ## [Parent/child with logstash 2.1.1](https://discuss.elastic.co/t/parent-child-with-logstash-2-1-1/37769)
**Author:** [@fchantrel](https://discuss.elastic.co/u/fchantrel)\ **Replies:** 4\ **Last updated:** [July 7, 2016, 8:48pm UTC](https://discuss.elastic.co/t/parent-child-with-logstash-2-1-1/37769 "2016-07-07T20:48:59Z")
Hi, i've tried to index child documents with logstash but it doesn't work since I use the 2.1.1 version. Here is my config : ''' ... filter { mutate { add\_field =\> { "parent" =\> "%{id}" } remove\_fie… --- ## [Plugin update schedule](https://discuss.elastic.co/t/plugin-update-schedule/54978)
**Author:** [@dad264](https://discuss.elastic.co/u/dad264)\ **Replies:** 0\ **Last updated:** [July 7, 2016, 5:26pm UTC](https://discuss.elastic.co/t/plugin-update-schedule/54978 "2016-07-07T17:26:02Z")
Is it best practice to regularly update all of the plugins in logstash? Specifically, after updating the base logstash install from (say) 2.3.1 to 2.3.3, should we then follow up with a logstash-plugin update to all of t… --- ## [Open Files Limit](https://discuss.elastic.co/t/open-files-limit/54961)
**Author:** [@jezeling](https://discuss.elastic.co/u/jezeling)\ **Replies:** 1\ **Last updated:** [July 7, 2016, 4:19pm UTC](https://discuss.elastic.co/t/open-files-limit/54961 "2016-07-07T16:19:25Z")
Hi, Apologies if this is in the wrong group but I have started to receive an open file error on my logstash nodes that push the data into my two node elastic cluster: - Reached open files limit: 4095, set by the 'max… --- ## [Not capturing syslog output \[SOLVED\]](https://discuss.elastic.co/t/not-capturing-syslog-output-solved/54735)
**Author:** [@Diggy](https://discuss.elastic.co/u/Diggy)\ **Replies:** 4\ **Last updated:** [July 7, 2016, 3:32pm UTC](https://discuss.elastic.co/t/not-capturing-syslog-output-solved/54735 "2016-07-07T15:32:52Z")
Hello, all. I have the latest ELK components installed on a CentOS 6 server, and running well. I'm currently capturing Windows logs, and CentOS 6 and 7 logs via Filebeat. However, I have a number of CentOS 5 boxes o… --- ## [Parse JSON string contained in a Syslog message](https://discuss.elastic.co/t/parse-json-string-contained-in-a-syslog-message/54951)
**Author:** [@norandom](https://discuss.elastic.co/u/norandom)\ **Replies:** 2\ **Last updated:** [July 7, 2016, 2:57pm UTC](https://discuss.elastic.co/t/parse-json-string-contained-in-a-syslog-message/54951 "2016-07-07T14:57:07Z")
Hi, I have a program which sends its JSON data via Syslog. Example: Jul 7 14:17:48 10.1.1.221 suricata\[28230\]: {"timestamp":"2016-07-07T14:19:52.081015+0000",..."} input { file { type =\> "sysl… --- ## [=\>"Unsupported Netflow version v10" warning using ipt-netflow](https://discuss.elastic.co/t/unsupported-netflow-version-v10-warning-using-ipt-netflow/54456)
**Author:** [@DRW](https://discuss.elastic.co/u/DRW)\ **Replies:** 4\ **Last updated:** [July 7, 2016, 2:12pm UTC](https://discuss.elastic.co/t/unsupported-netflow-version-v10-warning-using-ipt-netflow/54456 "2016-07-07T14:12:53Z")
Hi. I'm using ipt-netflow (https://github.com/aabc/ipt-netflow) to send netflow data at logstash. I am getting this. {:timestamp=\>"2016-06-30T17:26:06.068000+0000", :message=\>"Pipeline main started"} {:timestamp=\>"… --- ## [\[SOLVED\] Drop filter issue](https://discuss.elastic.co/t/solved-drop-filter-issue/54944)
**Author:** [@axelfelix](https://discuss.elastic.co/u/axelfelix)\ **Replies:** 4\ **Last updated:** [July 7, 2016, 1:51pm UTC](https://discuss.elastic.co/t/solved-drop-filter-issue/54944 "2016-07-07T13:51:24Z")
Hi all, I try to drop the field "message" in logstash: filter { if \[type\] == "test" { grok { match =\> {"message" =\> "%{TIMESTAMP\_ISO8601:syslog\_host\_time} %{IP:client} %{WORD:severity} %{… --- ## [Final multiline event not processed](https://discuss.elastic.co/t/final-multiline-event-not-processed/54887)
**Author:** [@leifmadsen](https://discuss.elastic.co/u/leifmadsen)\ **Replies:** 0\ **Last updated:** [July 7, 2016, 3:15am UTC](https://discuss.elastic.co/t/final-multiline-event-not-processed/54887 "2016-07-07T03:15:38Z")
So I'm using the multiline codec, and it seems to be working great. However, I'm doing filtering on the FileBeats side so that I'm only sending the multiline values that I expect to parse on the other end. (I can't use… --- ## [Logstash does not output to elastic , and now does not work at all](https://discuss.elastic.co/t/logstash-does-not-output-to-elastic-and-now-does-not-work-at-all/50912)
**Author:** [@gilisade](https://discuss.elastic.co/u/gilisade)\ **Replies:** 4\ **Last updated:** [July 6, 2016, 1:09pm UTC](https://discuss.elastic.co/t/logstash-does-not-output-to-elastic-and-now-does-not-work-at-all/50912 "2016-07-06T13:09:22Z")
hi , i've benn dealing with a problem for 2 days now i'm using logstash 2.3 on rhel VM my logstash configurations worked just fine until yesterday: i built another very simple configuration : input { file { … --- ## [Logstash Huge data import + Fast import](https://discuss.elastic.co/t/logstash-huge-data-import-fast-import/54909)
**Author:** [@Pierro](https://discuss.elastic.co/u/Pierro)\ **Replies:** 6\ **Last updated:** [July 7, 2016, 8:44am UTC](https://discuss.elastic.co/t/logstash-huge-data-import-fast-import/54909 "2016-07-07T08:44:23Z")
Hi, I'm a newbie with ELK and I've got some questions, of course. Let's try this one : My config: 3 servers: 1 master with Elasticsearch and logstash, 2 nodes with Elasticsearch. All are data store. The Logstash outpu… --- ## [Change the format of dates & calculate the difference](https://discuss.elastic.co/t/change-the-format-of-dates-calculate-the-difference/54899)
**Author:** [@Ravi\_Shanker\_Reddy](https://discuss.elastic.co/u/Ravi_Shanker_Reddy)\ **Replies:** 5\ **Last updated:** [July 7, 2016, 7:19am UTC](https://discuss.elastic.co/t/change-the-format-of-dates-calculate-the-difference/54899 "2016-07-07T07:19:18Z")
I am having a csv. In that I am having two times like this Wed May 18 23:00:01.000 2016 | Wed May 18 23:01:32.236 2016 I want to change this as date format and find the difference between them. I tried like this. But w… --- ## [Add field only if its does not exists?](https://discuss.elastic.co/t/add-field-only-if-its-does-not-exists/54894)
**Author:** [@memelet](https://discuss.elastic.co/u/memelet)\ **Replies:** 2\ **Last updated:** [July 7, 2016, 6:31am UTC](https://discuss.elastic.co/t/add-field-only-if-its-does-not-exists/54894 "2016-07-07T06:31:24Z")
I expect this is common, but I can't seem to find any info on how to set a field /only/ if it does not already exist. I have tried translate: translate { field =\> "level" dictionary =\> \[ "DEBUG", "DEBUG", … --- ## [Run Logstash using Java program](https://discuss.elastic.co/t/run-logstash-using-java-program/54877)
**Author:** [@wldbs0508](https://discuss.elastic.co/u/wldbs0508)\ **Replies:** 3\ **Last updated:** [July 7, 2016, 5:57am UTC](https://discuss.elastic.co/t/run-logstash-using-java-program/54877 "2016-07-07T05:57:56Z")
Hi, I would like to implement a program which runs Logstash with my custom config file. I wish to implement the program based on Java language. Is there anyone knows how to do it? Do I just run command like "./logstash… --- ## [How to add @timestamp to records that have no time at all](https://discuss.elastic.co/t/how-to-add-timestamp-to-records-that-have-no-time-at-all/54886)
**Author:** [@zoplex](https://discuss.elastic.co/u/zoplex)\ **Replies:** 4\ **Last updated:** [July 7, 2016, 2:09am UTC](https://discuss.elastic.co/t/how-to-add-timestamp-to-records-that-have-no-time-at-all/54886 "2016-07-07T02:09:57Z")
small percentage of incoming records have no time stamp ... how do I add that time stamp to the record - to make it equal to current run time? --- ## [Can long grok filter line be extended into multiple lines?](https://discuss.elastic.co/t/can-long-grok-filter-line-be-extended-into-multiple-lines/54880)
**Author:** [@zoplex](https://discuss.elastic.co/u/zoplex)\ **Replies:** 7\ **Last updated:** [July 7, 2016, 12:51am UTC](https://discuss.elastic.co/t/can-long-grok-filter-line-be-extended-into-multiple-lines/54880 "2016-07-07T00:51:23Z")
is there a continuation syntax - some of my filters are stretching beyond 150 characters - would be nice if that could be extended into second line ... Thanks, --- ## [Logstash ruby filter not working](https://discuss.elastic.co/t/logstash-ruby-filter-not-working/54807)
**Author:** [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Replies:** 3\ **Last updated:** [July 6, 2016, 9:09pm UTC](https://discuss.elastic.co/t/logstash-ruby-filter-not-working/54807 "2016-07-06T21:09:43Z")
Hi Team, I am trying to get the number from the underscore in the \_id, so I tried using ruby filter to get the number below is the following details, Sample post data, POST test\_item/item/149927\_1 { "ITEM\_ID": 1… --- ## [Logstash agents without Go (CentOS 5)](https://discuss.elastic.co/t/logstash-agents-without-go-centos-5/54856)
**Author:** [@smilin\_stan](https://discuss.elastic.co/u/smilin_stan)\ **Replies:** 2\ **Last updated:** [July 6, 2016, 9:01pm UTC](https://discuss.elastic.co/t/logstash-agents-without-go-centos-5/54856 "2016-07-06T21:01:29Z")
I have to collect log files from a number of servers that are running CentOS 5 which has no support for Go, which means no FileBeat. Is there a tool for collecting/shipping log files that isn't written in Go? Thanks … --- ## [Cannot get winlogbeats into logstash](https://discuss.elastic.co/t/cannot-get-winlogbeats-into-logstash/54787)
**Author:** [@Dalton\_Culp](https://discuss.elastic.co/u/Dalton_Culp)\ **Replies:** 11\ **Last updated:** [July 6, 2016, 8:57pm UTC](https://discuss.elastic.co/t/cannot-get-winlogbeats-into-logstash/54787 "2016-07-06T20:57:51Z")
i can show pictures of configurations if people need them. i think what is happening is elastic search is looking at loopback instead of eth0 --- ## [Range Filter - Accessing Nested Fields](https://discuss.elastic.co/t/range-filter-accessing-nested-fields/54418)
**Author:** [@DJRickyB](https://discuss.elastic.co/u/DJRickyB)\ **Replies:** 2\ **Last updated:** [July 6, 2016, 8:53pm UTC](https://discuss.elastic.co/t/range-filter-accessing-nested-fields/54418 "2016-07-06T20:53:42Z")
Given the syntax example for the ranges parameter of the range plugin as follows, how would you access/evaluate the value of a nested field? filter { range { ranges =\> \[ "message", 0, 10, "tag:short", … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1129) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1131)