# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1231 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1232 --- ## [Updating template to include DocValue via api](https://discuss.elastic.co/t/updating-template-to-include-docvalue-via-api/31659)
**Author:** [@lhorsky1](https://discuss.elastic.co/u/lhorsky1)\ **Replies:** 4\ **Last updated:** [October 12, 2015, 5:13pm UTC](https://discuss.elastic.co/t/updating-template-to-include-docvalue-via-api/31659 "2015-10-12T17:13:40Z")
I want to add DocValue to my default logstash-\* template. I have seen documentation that says to add it to the raw section of the not\_analyzed. But I don't have a config file on the server, so I will need to add this usi… --- ## [Reverse connections from lumberjack](https://discuss.elastic.co/t/reverse-connections-from-lumberjack/31990)
**Author:** [@qwerty](https://discuss.elastic.co/u/qwerty)\ **Replies:** 5\ **Last updated:** [October 12, 2015, 5:12pm UTC](https://discuss.elastic.co/t/reverse-connections-from-lumberjack/31990 "2015-10-12T17:12:46Z")
Here is my setup: 1) system-A -- running logstash-forwarder, shipping to B 2) system-B -- running lumberjack on port 5043 and logstash-forwarder 3) system-C -- running logstash-forwarder, shipping to B I'm seeing rever… --- ## [Kafka input not resuming where it stopped](https://discuss.elastic.co/t/kafka-input-not-resuming-where-it-stopped/31882)
**Author:** [@mihir\_ray](https://discuss.elastic.co/u/mihir_ray)\ **Replies:** 6\ **Last updated:** [October 12, 2015, 1:13pm UTC](https://discuss.elastic.co/t/kafka-input-not-resuming-where-it-stopped/31882 "2015-10-12T13:13:47Z")
HI, I have a kafka cluster from where i am reading data and doing my processing. The input looks like: input { kafka { topic\_id =\> "sb\_logs" codec =\> "plain" zk\_connect =\> "cpanaetl01.sli… --- ## [Stomp input does not reconnect on network partition](https://discuss.elastic.co/t/stomp-input-does-not-reconnect-on-network-partition/28317)
**Author:** [@Johntdyer](https://discuss.elastic.co/u/Johntdyer)\ **Replies:** 1\ **Last updated:** [October 12, 2015, 12:59am UTC](https://discuss.elastic.co/t/stomp-input-does-not-reconnect-on-network-partition/28317 "2015-10-12T00:59:47Z")
Anyone else noticed that when there is a network partition Logstash Stomp does not reconnect to the broker ? When this happens I am forced to restart indexers which is of course a pain... I am curious if anyone else h… --- ## [Memory issue with logstash?](https://discuss.elastic.co/t/memory-issue-with-logstash/31966)
**Author:** [@sampasei](https://discuss.elastic.co/u/sampasei)\ **Replies:** 1\ **Last updated:** [October 11, 2015, 9:05pm UTC](https://discuss.elastic.co/t/memory-issue-with-logstash/31966 "2015-10-11T21:05:16Z")
Hi all, i've configured logstash in order to take some log files (these files are never updated because are made ony for tests) and write record data on Elastichsearch and Mongo. Below the conf file input { file{ … --- ## [Nand and xor operators not working](https://discuss.elastic.co/t/nand-and-xor-operators-not-working/31972)
**Author:** [@RajkumarV](https://discuss.elastic.co/u/RajkumarV)\ **Replies:** 1\ **Last updated:** [October 11, 2015, 7:34pm UTC](https://discuss.elastic.co/t/nand-and-xor-operators-not-working/31972 "2015-10-11T19:34:49Z")
Hi I am using logstash 1.5.4 version on Ubuntu 14.04. I am unable to make use of nand, xor operator in conditionals. The error resulted and the configuration used for xor operator example is given below: (similar e… --- ## [Cannot input log with regular expression](https://discuss.elastic.co/t/cannot-input-log-with-regular-expression/31946)
**Author:** [@M.Tyler](https://discuss.elastic.co/u/M.Tyler)\ **Replies:** 2\ **Last updated:** [October 11, 2015, 6:25am UTC](https://discuss.elastic.co/t/cannot-input-log-with-regular-expression/31946 "2015-10-11T06:25:57Z")
Please let me question about regular expression for logstash Logstash version:1.5.4 I'd like to input MPLog-xxxxxxxx-xxxxxx.log that is log of Windows Defender. (There is C:\\ProgramData\\Microsoft\\Windows Defender\\Sup… --- ## [Input-S3 Plugin - Logic flaw with sincedb file](https://discuss.elastic.co/t/input-s3-plugin-logic-flaw-with-sincedb-file/31956)
**Author:** [@davidski](https://discuss.elastic.co/u/davidski)\ **Replies:** 0\ **Last updated:** [October 10, 2015, 3:08pm UTC](https://discuss.elastic.co/t/input-s3-plugin-logic-flaw-with-sincedb-file/31956 "2015-10-10T15:08:33Z")
I'm having some problems with the input-s3 plugin missing files in my bucket and causing my entire pipeline to stall out (I'm doing batched processing and use the emptying of the source bucket as an indicator of being fi… --- ## [Keep hierarchy in json data](https://discuss.elastic.co/t/keep-hierarchy-in-json-data/31881)
**Author:** [@Ducheol\_Kim](https://discuss.elastic.co/u/Ducheol_Kim)\ **Replies:** 4\ **Last updated:** [October 9, 2015, 6:34pm UTC](https://discuss.elastic.co/t/keep-hierarchy-in-json-data/31881 "2015-10-09T18:34:21Z")
Hi All. I try to parse the hierarchical json data. Problem is there are same field in different level like below. {"id":"TEST","level2":{"id":"Level2 id"}} In above case, final appeared id is overwrite previous one a… --- ## [\[Help Request\] HTTP Input with Multiline JSON Post Data](https://discuss.elastic.co/t/help-request-http-input-with-multiline-json-post-data/31920)
**Author:** [@tmo](https://discuss.elastic.co/u/tmo)\ **Replies:** 1\ **Last updated:** [October 9, 2015, 5:59pm UTC](https://discuss.elastic.co/t/help-request-http-input-with-multiline-json-post-data/31920 "2015-10-09T17:59:12Z")
I'm trying to parse a newline separated HTTPS POST data feed, each line is a JSON object. I also have an output to elastic search. Right now I'm trying to use the json\_lines codec and that doesn't seem to be working a… --- ## ["Extracting" Data from a Message](https://discuss.elastic.co/t/extracting-data-from-a-message/31795)
**Author:** [@just5guy](https://discuss.elastic.co/u/just5guy)\ **Replies:** 3\ **Last updated:** [October 9, 2015, 4:02pm UTC](https://discuss.elastic.co/t/extracting-data-from-a-message/31795 "2015-10-09T16:02:31Z")
So I am coming from Splunk, currently using Graylog, and trying to set up the ELK Stack to analyze power usage. I have my Energy Monitor sending messages to the server properly. Here are a couple lines for an example: … --- ## [Logstash parsing two days behind](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879)
**Author:** [@kmroz](https://discuss.elastic.co/u/kmroz)\ **Replies:** 13\ **Last updated:** [October 9, 2015, 3:41pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879 "2015-10-09T15:41:45Z")
Hello, I'm forwarding logs from a remote server to my docker container containing the elk stack and after looking at the logs, it seems they are two days behind. Any reason why this is? Is there a way to fix this? Thank… --- ## [Comparing two string in ruby filter / comparing two string fields](https://discuss.elastic.co/t/comparing-two-string-in-ruby-filter-comparing-two-string-fields/31914)
**Author:** [@Navneet\_Mathpal](https://discuss.elastic.co/u/Navneet_Mathpal)\ **Replies:** 2\ **Last updated:** [October 9, 2015, 2:45pm UTC](https://discuss.elastic.co/t/comparing-two-string-in-ruby-filter-comparing-two-string-fields/31914 "2015-10-09T14:45:40Z")
Hi , I was trying to compare two string in logstash ruby filter inside code , it it always showing wrong result . same if I am trying in ruby IDE is working perfectly fine . I am trying to do something code =\> " … --- ## [\[SOLVED\] Elasticsearch output filter by tags... now failing when on new indices](https://discuss.elastic.co/t/solved-elasticsearch-output-filter-by-tags-now-failing-when-on-new-indices/31884)
**Author:** [@alexolivan](https://discuss.elastic.co/u/alexolivan)\ **Replies:** 1\ **Last updated:** [October 9, 2015, 2:44pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-output-filter-by-tags-now-failing-when-on-new-indices/31884 "2015-10-09T14:44:13Z")
Hi forum... I'm puzzled by several issues here... but this one is notably strange: Im injecting to elasticsearch different indices filtering by tag... This has been working well (till now) heres my output filter file t… --- ## [Kafka Input plugin error](https://discuss.elastic.co/t/kafka-input-plugin-error/31450)
**Author:** [@astro](https://discuss.elastic.co/u/astro)\ **Replies:** 7\ **Last updated:** [October 9, 2015, 1:30pm UTC](https://discuss.elastic.co/t/kafka-input-plugin-error/31450 "2015-10-09T13:30:36Z")
Hi All, I have setup Kafka and trying to use kafka input and output plugin For Higher availability I tried to use multiple zookeeper and multiple brokers all setup on different machines . zk1:2181 , zk2:2181, zk3:2181… --- ## [Logstash - CSV to JSON](https://discuss.elastic.co/t/logstash-csv-to-json/31826)
**Author:** [@schilwan](https://discuss.elastic.co/u/schilwan)\ **Replies:** 7\ **Last updated:** [October 9, 2015, 11:27am UTC](https://discuss.elastic.co/t/logstash-csv-to-json/31826 "2015-10-09T11:27:44Z")
Hopefully this is the right place, I am currently fairly new to the ELK stack so not sure if what I am trying to do in logstash is feasible. I am consuming a CSV file and want to convert it into a JSON format as followe… --- ## [\[solved\] Logstash-Forwarder, json codec and multiline issue](https://discuss.elastic.co/t/solved-logstash-forwarder-json-codec-and-multiline-issue/31824)
**Author:** [@jeroentbt](https://discuss.elastic.co/u/jeroentbt)\ **Replies:** 4\ **Last updated:** [October 9, 2015, 9:29am UTC](https://discuss.elastic.co/t/solved-logstash-forwarder-json-codec-and-multiline-issue/31824 "2015-10-09T09:29:51Z")
Hey, I'm seeing a lot of issues with certain specific multiline log entries. Here's a log entry: D, \[2015-10-07T16:11:53.434110 #20457\] DEBUG -- : Error occurred while parsing request parameters. Contents: { "payl… --- ## [Remove message on filter before inserting in ES](https://discuss.elastic.co/t/remove-message-on-filter-before-inserting-in-es/31776)
**Author:** [@ashwin2011](https://discuss.elastic.co/u/ashwin2011)\ **Replies:** 3\ **Last updated:** [October 9, 2015, 7:26am UTC](https://discuss.elastic.co/t/remove-message-on-filter-before-inserting-in-es/31776 "2015-10-09T07:26:25Z")
Hi, Here is my actual config and output : filter { if \[type\] == "ufdb" { grok { match =\> \[ "message", "%{TIMESTAMP\_ISO8601:logdate}%{SPACE}%{DATA}%{SPACE}%{WORD:status}%{SPACE}… --- ## [Logstash output ~~ Logstash input](https://discuss.elastic.co/t/logstash-output-logstash-input/31887)
**Author:** [@treehouse](https://discuss.elastic.co/u/treehouse)\ **Replies:** 1\ **Last updated:** [October 9, 2015, 2:58am UTC](https://discuss.elastic.co/t/logstash-output-logstash-input/31887 "2015-10-09T02:58:01Z")
Hi Our architecture: One logstash instance accepting all of the log-courier shippers from N machines. Output from this logstash into two unique logstash instances, \> stage and \> production. Each of these instances… --- ## [Logstash with Redis](https://discuss.elastic.co/t/logstash-with-redis/29474)
**Author:** [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)\ **Replies:** 4\ **Last updated:** [October 8, 2015, 8:27pm UTC](https://discuss.elastic.co/t/logstash-with-redis/29474 "2015-10-08T20:27:13Z")
I have an ELK stack with redis as my queue. Currently I am running one redis instance on my sensors and it reports back to a central redis "cluster". I put cluster in quotes because I don't have it set up as a cluster … --- ## [Logstash parsing not consistant](https://discuss.elastic.co/t/logstash-parsing-not-consistant/31732)
**Author:** [@l1carter](https://discuss.elastic.co/u/l1carter)\ **Replies:** 5\ **Last updated:** [October 8, 2015, 7:46pm UTC](https://discuss.elastic.co/t/logstash-parsing-not-consistant/31732 "2015-10-08T19:46:15Z")
New here and new to ELK. I'm to a point however that I think it may be easier to ask as someone may have seen this or know the issue... two identical logs come into the system (file input type)... one line matches my fil… --- ## [Error writing to pipe, closing pipe](https://discuss.elastic.co/t/error-writing-to-pipe-closing-pipe/31863)
**Author:** [@Navneet\_Mathpal](https://discuss.elastic.co/u/Navneet_Mathpal)\ **Replies:** 1\ **Last updated:** [October 8, 2015, 7:40pm UTC](https://discuss.elastic.co/t/error-writing-to-pipe-closing-pipe/31863 "2015-10-08T19:40:03Z")
Hi , I am trying to call a java file using logstash in output using pipe . input { stdin {} } filter { ruby { code =\> " event\['id'\]=File.open('/elkt/software/bkup/logstash-1.5.4/bin/wid.id', &:readline) " … --- ## [Issues with grok parsing logs that vary](https://discuss.elastic.co/t/issues-with-grok-parsing-logs-that-vary/31853)
**Author:** [@dlb1001](https://discuss.elastic.co/u/dlb1001)\ **Replies:** 3\ **Last updated:** [October 8, 2015, 4:21pm UTC](https://discuss.elastic.co/t/issues-with-grok-parsing-logs-that-vary/31853 "2015-10-08T16:21:56Z")
Hello, I'm having issues with my logstash grok parsing. I wish the logs were more uniform but since we have a proxy in place, I can expect from 1-4 ip addresses in the apache log line. Also what makes it difficult is… --- ## [Filter by port using pipe directive](https://discuss.elastic.co/t/filter-by-port-using-pipe-directive/31688)
**Author:** [@Ben19](https://discuss.elastic.co/u/Ben19)\ **Replies:** 4\ **Last updated:** [October 8, 2015, 2:40pm UTC](https://discuss.elastic.co/t/filter-by-port-using-pipe-directive/31688 "2015-10-08T14:40:47Z")
Hello, I’m attempting to collect sflow logs from a number of exporters that will be exporting and sending the traffic on UDP port 6434. I’m using the pipe directive since I can specify the command setting, otherwise I’… --- ## [Some fields missing](https://discuss.elastic.co/t/some-fields-missing/31828)
**Author:** [@Anmol\_Gupta](https://discuss.elastic.co/u/Anmol_Gupta)\ **Replies:** 5\ **Last updated:** [October 8, 2015, 11:36am UTC](https://discuss.elastic.co/t/some-fields-missing/31828 "2015-10-08T11:36:01Z")
when i am indexing my log file from logstash some fields are missing . \_grokparsefailure is coming . I am indexing a log file it has log levels so my error and trace are getting indexed but warn , debug are missing and w… --- ## [Logstash aggregation filter configuration](https://discuss.elastic.co/t/logstash-aggregation-filter-configuration/31814)
**Author:** [@grycuk](https://discuss.elastic.co/u/grycuk)\ **Replies:** 2\ **Last updated:** [October 8, 2015, 6:24am UTC](https://discuss.elastic.co/t/logstash-aggregation-filter-configuration/31814 "2015-10-08T06:24:46Z")
Hi there I'm hoping someone can help me with this please. If I have an input file in json format with a field named uuid that I wish to use as a task id in the logstash aggregate plugin. What is the correct syntax to … --- ## [Wrong ElasticSearch template being used](https://discuss.elastic.co/t/wrong-elasticsearch-template-being-used/31812)
**Author:** [@Dualnames](https://discuss.elastic.co/u/Dualnames)\ **Replies:** 5\ **Last updated:** [October 8, 2015, 2:19am UTC](https://discuss.elastic.co/t/wrong-elasticsearch-template-being-used/31812 "2015-10-08T02:19:46Z")
I am not sure if this is the correct forum for this but I am having issues with one of my logstash configurations when it is sent to elasticsearch. I am brand new to ELK and still learning how everything works. So far, … --- ## [Remain json data as field in Logstash](https://discuss.elastic.co/t/remain-json-data-as-field-in-logstash/31784)
**Author:** [@Ducheol\_Kim](https://discuss.elastic.co/u/Ducheol_Kim)\ **Replies:** 3\ **Last updated:** [October 7, 2015, 7:44pm UTC](https://discuss.elastic.co/t/remain-json-data-as-field-in-logstash/31784 "2015-10-07T19:44:09Z")
Hi Guys. Currently, I'm building component which log json data in ES. So I'm trying to use below grok filter pattern to parse , but it isn't work. Input Tue, 06 Oct 2015 20:04:15 +0000 - http://localhost/testurl?id… --- ## [Memory usage on startup](https://discuss.elastic.co/t/memory-usage-on-startup/31789)
**Author:** [@ofer\_peretz](https://discuss.elastic.co/u/ofer_peretz)\ **Replies:** 1\ **Last updated:** [October 7, 2015, 5:14pm UTC](https://discuss.elastic.co/t/memory-usage-on-startup/31789 "2015-10-07T17:14:45Z")
Hi, I have using logstash forwarder to ship log from lightweight machine with limited RAM. When logstash forwarder is running on the machine it uses 200M when starting it without processing any logs yet. My question i… --- ## [File input not working - again](https://discuss.elastic.co/t/file-input-not-working-again/31773)
**Author:** [@Fabio\_Rocha](https://discuss.elastic.co/u/Fabio_Rocha)\ **Replies:** 2\ **Last updated:** [October 7, 2015, 2:05pm UTC](https://discuss.elastic.co/t/file-input-not-working-again/31773 "2015-10-07T14:05:02Z")
Hello folks, I've got issues when tried to read syslog-ng files with file input. It simply doesn't work. Here is my config: input { file { path =\> \[ "/var/log/10-SWITCH/\*" \] … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1230) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1232)