# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1031 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1032 --- ## [Event.get("message") & http input - where is the message?](https://discuss.elastic.co/t/event-get-message-http-input-where-is-the-message/76000)
**Author:** [@NirKa](https://discuss.elastic.co/u/NirKa)\ **Replies:** 1\ **Last updated:** [February 23, 2017, 6:44am UTC](https://discuss.elastic.co/t/event-get-message-http-input-where-is-the-message/76000 "2017-02-23T06:44:49Z")
Hi, I'm creating a new plugin that writes events to google pub sub. I don't understand the behavior of "message" when using http input. I tried using stdin input & I can see the content of the message when I execute … --- ## [Renaming dynamic nested json message fields](https://discuss.elastic.co/t/renaming-dynamic-nested-json-message-fields/75799)
**Author:** [@pbData](https://discuss.elastic.co/u/pbData)\ **Replies:** 1\ **Last updated:** [February 23, 2017, 6:39am UTC](https://discuss.elastic.co/t/renaming-dynamic-nested-json-message-fields/75799 "2017-02-23T06:39:24Z")
I have the following json data { "parent" : { "child\_1" : { "field\_1":123, "field\_2":"my\_child\_1\_string" }, "child\_2" : { "field\_1":456, "field\_2":"my\_child\_2\_string" } } } I need to be able to … --- ## [Index multiple csvs](https://discuss.elastic.co/t/index-multiple-csvs/74933)
**Author:** [@Tatireddy\_Reddy](https://discuss.elastic.co/u/Tatireddy_Reddy)\ **Replies:** 3\ **Last updated:** [February 23, 2017, 6:37am UTC](https://discuss.elastic.co/t/index-multiple-csvs/74933 "2017-02-23T06:37:45Z")
Hi All, I have mulitiple csvs with different columns how do i need to index using logstash instead of creating one big csv. Appreciated help. --- ## [When I active persisten connection in logstash it through out Error](https://discuss.elastic.co/t/when-i-active-persisten-connection-in-logstash-it-through-out-error/76157)
**Author:** [@jamal](https://discuss.elastic.co/u/jamal)\ **Replies:** 0\ **Last updated:** [February 23, 2017, 6:18am UTC](https://discuss.elastic.co/t/when-i-active-persisten-connection-in-logstash-it-through-out-error/76157 "2017-02-23T06:18:43Z")
I heve the following configuration in logstash config file pipeline.batch.size: 2 pipeline.batch.delay: 20 queue.type: persisted queue.max\_events: 0 queue.max\_bytes: 2gb queue.checkpoint.acks: 2024 queue.checkpoint… --- ## [Can't get Logstash to read existing Kafka topic from start](https://discuss.elastic.co/t/cant-get-logstash-to-read-existing-kafka-topic-from-start/73306)
**Author:** [@ankh](https://discuss.elastic.co/u/ankh)\ **Replies:** 7\ **Last updated:** [February 23, 2017, 4:50am UTC](https://discuss.elastic.co/t/cant-get-logstash-to-read-existing-kafka-topic-from-start/73306 "2017-02-23T04:50:22Z")
I'm trying to consume a Kafka topic using Logstash, for indexing by Elasticsearch. The Kafka events are JSON documents. We recently upgraded our Elastic Stack to 5.1.2. I believe that I was able to consume the topic … --- ## [Mapper for \[@timestamp\] conflicts](https://discuss.elastic.co/t/mapper-for-timestamp-conflicts/76144)
**Author:** [@rsaeks](https://discuss.elastic.co/u/rsaeks)\ **Replies:** 0\ **Last updated:** [February 23, 2017, 4:12am UTC](https://discuss.elastic.co/t/mapper-for-timestamp-conflicts/76144 "2017-02-23T04:12:41Z")
After trying to track down why some data wasn't showing up, I found the following warning message in my logstash file: \[2017-02-22T21:34:23,198\]\[WARN \]\[logstash.outputs.elasticsearch\] Failed action. {:status=\>400, :act… --- ## [Logstash grok filter](https://discuss.elastic.co/t/logstash-grok-filter/75813)
**Author:** [@RSundareswaran](https://discuss.elastic.co/u/RSundareswaran)\ **Replies:** 2\ **Last updated:** [February 23, 2017, 2:01am UTC](https://discuss.elastic.co/t/logstash-grok-filter/75813 "2017-02-23T02:01:02Z")
I have a log file which has a date time in 'yyyyMMdd\_HHmmss\_SSS' format. I am successful in parsing this with \_ as delimiter and getting as 3 different text field in ES. But I need this to be converted as ISO\_8601 so I c… --- ## [Docker Logstash initialisation error](https://discuss.elastic.co/t/docker-logstash-initialisation-error/75972)
**Author:** [@pravinnair](https://discuss.elastic.co/u/pravinnair)\ **Replies:** 5\ **Last updated:** [February 23, 2017, 12:50am UTC](https://discuss.elastic.co/t/docker-logstash-initialisation-error/75972 "2017-02-23T00:50:47Z")
Hi I recive this error in logstash when i try to start it. This stops the poller.Any help or input can help. \[pool-2-thread-1\] DEBUG logstash.instrument.periodicpoller.cgroup - Error, cannot retrieve cgroups informa… --- ## [Logstash JSON syslog\_message parsing](https://discuss.elastic.co/t/logstash-json-syslog-message-parsing/75526)
**Author:** [@blakefallow](https://discuss.elastic.co/u/blakefallow)\ **Replies:** 2\ **Last updated:** [February 23, 2017, 12:08am UTC](https://discuss.elastic.co/t/logstash-json-syslog-message-parsing/75526 "2017-02-23T00:08:40Z")
OS: Ubuntu 16.04 Logstash: 5.2.1 Elasticsearch: 5.1 I've configured our Deis platform to send logs to our Logstack node with no issues. However, I'm still new to Ruby and Logstash is not yet my strong suit. I can't see… --- ## [Not able to import complete data using Logstash to Elasticsearch](https://discuss.elastic.co/t/not-able-to-import-complete-data-using-logstash-to-elasticsearch/75825)
**Author:** [@Varun\_Patel](https://discuss.elastic.co/u/Varun_Patel)\ **Replies:** 4\ **Last updated:** [February 22, 2017, 8:11pm UTC](https://discuss.elastic.co/t/not-able-to-import-complete-data-using-logstash-to-elasticsearch/75825 "2017-02-22T20:11:58Z")
Hi, I am using Logstash with JDBC connection for importing data from database to elasticsearch. When i am importing data, all the data present in database is not imported. There are 487745 records in my database but w… --- ## [Logstash not seeing conf files in conf.d dir](https://discuss.elastic.co/t/logstash-not-seeing-conf-files-in-conf-d-dir/75931)
**Author:** [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)\ **Replies:** 35\ **Last updated:** [February 22, 2017, 6:26pm UTC](https://discuss.elastic.co/t/logstash-not-seeing-conf-files-in-conf-d-dir/75931 "2017-02-22T18:26:02Z")
I am not new to logstash but I am to 5.X My issue is with files in the /etc/logstash/conf.d directory. When I created my configuration file in this directory and start logstash as a daemon it does nothing. When I sta… --- ## [Building First Filter Plugin](https://discuss.elastic.co/t/building-first-filter-plugin/76084)
**Author:** [@dishmael](https://discuss.elastic.co/u/dishmael)\ **Replies:** 1\ **Last updated:** [February 22, 2017, 5:00pm UTC](https://discuss.elastic.co/t/building-first-filter-plugin/76084 "2017-02-22T17:00:39Z")
I am following along with the How to write a Logstash filter plugin, but when I get to the section for building, I'm getting an error: $ build install Fetching git://github.com/elastic/logstash.git Fetching git://githu… --- ## [Fresh logstash 5.2 install gives log4j warnings and no logging from plugins](https://discuss.elastic.co/t/fresh-logstash-5-2-install-gives-log4j-warnings-and-no-logging-from-plugins/75530)
**Author:** [@Carlos\_Alexandre](https://discuss.elastic.co/u/Carlos_Alexandre)\ **Replies:** 5\ **Last updated:** [February 22, 2017, 3:04pm UTC](https://discuss.elastic.co/t/fresh-logstash-5-2-install-gives-log4j-warnings-and-no-logging-from-plugins/75530 "2017-02-22T15:04:44Z")
I just downloaded the new logstash version (5.2.1), and after starting it I noticed a warning message saying that log4j needs to be initialized properly. I also noticed that plugins like kafka, beats, etc, were not emitt… --- ## [Logstash filter: updating nested document event by querying ES](https://discuss.elastic.co/t/logstash-filter-updating-nested-document-event-by-querying-es/76046)
**Author:** [@shravanthid](https://discuss.elastic.co/u/shravanthid)\ **Replies:** 0\ **Last updated:** [February 22, 2017, 2:09pm UTC](https://discuss.elastic.co/t/logstash-filter-updating-nested-document-event-by-querying-es/76046 "2017-02-22T14:09:56Z")
Hi, I have a requirement where I need to join 2 indexes "test-a-2017-02-22" index doc structure: { "event": { "a": "A", "b": "B" } } "test-b" doc structure: { "event": { … --- ## [Multiple Output with Slow and Fast Receiving End](https://discuss.elastic.co/t/multiple-output-with-slow-and-fast-receiving-end/76010)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 1\ **Last updated:** [February 22, 2017, 2:28pm UTC](https://discuss.elastic.co/t/multiple-output-with-slow-and-fast-receiving-end/76010 "2017-02-22T14:28:54Z")
In a situation where I have 2 or more output e.g. an elasticsearch and a kafka. If kafka is very slow and elasticsearch is very fast. What will happen to the reading speed of logstash? Will logstash wait for kafka befor… --- ## [Parse optional field with or without leading spaces](https://discuss.elastic.co/t/parse-optional-field-with-or-without-leading-spaces/76041)
**Author:** [@mflash](https://discuss.elastic.co/u/mflash)\ **Replies:** 0\ **Last updated:** [February 22, 2017, 1:36pm UTC](https://discuss.elastic.co/t/parse-optional-field-with-or-without-leading-spaces/76041 "2017-02-22T13:36:16Z")
hello, i am trying to parse the following log messages with grok: \[2017-02-21T16:20:11.709Z\] \[debug\] \[mosaicresult\] publish mosaicresult client id: gIsdfdsfgvdsfvdfs channel: localMosaicChannel \[{"ffvfv":"dfvfdvdfv"… --- ## [Changing the date Filter to get exact same timestamp in kibana as in log](https://discuss.elastic.co/t/changing-the-date-filter-to-get-exact-same-timestamp-in-kibana-as-in-log/76009)
**Author:** [@anisen](https://discuss.elastic.co/u/anisen)\ **Replies:** 3\ **Last updated:** [February 22, 2017, 1:25pm UTC](https://discuss.elastic.co/t/changing-the-date-filter-to-get-exact-same-timestamp-in-kibana-as-in-log/76009 "2017-02-22T13:25:14Z")
Hi, My log looks like below: www.xyz.com 100.181.12.41 - - \[04/Dec/2013:07:35:39 -0600\] "GET /abc/images/someimage.png HTTP/1.1" 200 My Grok filter is as below: grok { match =\> { "message" =\> '%{NOTSPACE:… --- ## [Can't get past first base with Logstash](https://discuss.elastic.co/t/cant-get-past-first-base-with-logstash/76034)
**Author:** [@Joe4](https://discuss.elastic.co/u/Joe4)\ **Replies:** 1\ **Last updated:** [February 22, 2017, 12:57pm UTC](https://discuss.elastic.co/t/cant-get-past-first-base-with-logstash/76034 "2017-02-22T12:57:33Z")
Everything fine and dandy with ELK otherwise - Filebeat, Elasticsearch, Kibana seems to work as expected. But can't get logstash to do this first thing. Centos 7.3 64 bit - tried installing through repository then foll… --- ## [Anyone get logstash-input-xlsx or logstash-codec-excel work?](https://discuss.elastic.co/t/anyone-get-logstash-input-xlsx-or-logstash-codec-excel-work/55448)
**Author:** [@zotsing](https://discuss.elastic.co/u/zotsing)\ **Replies:** 1\ **Last updated:** [February 22, 2017, 12:03pm UTC](https://discuss.elastic.co/t/anyone-get-logstash-input-xlsx-or-logstash-codec-excel-work/55448 "2017-02-22T12:03:00Z")
Tried for quite a while, but keep getting the following message, hope someone can help me to get it work: Message: failed to open #@filestat=#, @state\_history=\[:watched\], @ignore\_older=nil, @close\_older=nil, @initial=t… --- ## [Logstash Plugin stop woriking after one day](https://discuss.elastic.co/t/logstash-plugin-stop-woriking-after-one-day/75818)
**Author:** [@jamal](https://discuss.elastic.co/u/jamal)\ **Replies:** 3\ **Last updated:** [February 22, 2017, 11:01am UTC](https://discuss.elastic.co/t/logstash-plugin-stop-woriking-after-one-day/75818 "2017-02-22T11:01:23Z")
I am using input-rss and input-twitter with output-elasticsearh whe run logstash for one day all the plugin work fine input-twitter and input-rss get the data but after one process is running but plugin dont get the da… --- ## [Jdbc plugin : column with xml content](https://discuss.elastic.co/t/jdbc-plugin-column-with-xml-content/75906)
**Author:** [@Lemec\_Cinq](https://discuss.elastic.co/u/Lemec_Cinq)\ **Replies:** 2\ **Last updated:** [February 22, 2017, 10:59am UTC](https://discuss.elastic.co/t/jdbc-plugin-column-with-xml-content/75906 "2017-02-22T10:59:42Z")
I am importing a MS SQL table with an column that contains xml those are values example: \ \ \ \ ..… --- ## [\_grokparsefailure tag - fails to do a thing](https://discuss.elastic.co/t/-grokparsefailure-tag-fails-to-do-a-thing/76005)
**Author:** [@yami](https://discuss.elastic.co/u/yami)\ **Replies:** 2\ **Last updated:** [February 22, 2017, 9:20am UTC](https://discuss.elastic.co/t/-grokparsefailure-tag-fails-to-do-a-thing/76005 "2017-02-22T09:20:27Z")
i tried to make grok pattern: PODPISY (?:\[\\s\\S\]\*?\<\\/podpisy\>) it's based on REGEX: and it works as it should. I use it in config like that: filter{ grok { match =\> { "message" =\> "%{PODPISY}" }} } A log a… --- ## [Winlogbeat -\> Logstash -\> JsonFile -\> Logstash -\> EL](https://discuss.elastic.co/t/winlogbeat-logstash-jsonfile-logstash-el/76012)
**Author:** [@Alo](https://discuss.elastic.co/u/Alo)\ **Replies:** 0\ **Last updated:** [February 22, 2017, 10:06am UTC](https://discuss.elastic.co/t/winlogbeat-logstash-jsonfile-logstash-el/76012 "2017-02-22T10:06:43Z")
Hi there:) I'm having a scenario where we are grabbing Events via Winlogbeat which writes them to a RAID 1 NAS for backup reasons. Unfortunately we can't let winlogbeat write directly in EL. first logstash cfg: input… --- ## [Logstash | Apache Acces log | Parsing](https://discuss.elastic.co/t/logstash-apache-acces-log-parsing/75822)
**Author:** [@aliahsan81](https://discuss.elastic.co/u/aliahsan81)\ **Replies:** 4\ **Last updated:** [February 22, 2017, 9:36am UTC](https://discuss.elastic.co/t/logstash-apache-acces-log-parsing/75822 "2017-02-22T09:36:14Z")
Hi All, I have custom apache log format, I am struggling to make it parse using logstash. Can any one help me. LogFormat "%{X-Forwarded-For}i %h %l %u %t \\"%r\\" %\>s %b \\"%{Referer}i\\" \\"%{User-Agent}i\\" \\"%{True-Clie… --- ## [Error installing logstash output plugin](https://discuss.elastic.co/t/error-installing-logstash-output-plugin/75391)
**Author:** [@Lorenc\_Hysi](https://discuss.elastic.co/u/Lorenc_Hysi)\ **Replies:** 1\ **Last updated:** [February 22, 2017, 9:19am UTC](https://discuss.elastic.co/t/error-installing-logstash-output-plugin/75391 "2017-02-22T09:19:04Z")
Hello all, I'm having a problem install the syslog output plugin to logstash 5.2. After installing logstash using yum from a local rpm. I run this command: bin/logstash-plugin install logstash-output-syslog Howeve… --- ## [Remove blank fields from output file](https://discuss.elastic.co/t/remove-blank-fields-from-output-file/75886)
**Author:** [@Tarun\_Kalluri](https://discuss.elastic.co/u/Tarun_Kalluri)\ **Replies:** 3\ **Last updated:** [February 22, 2017, 9:05am UTC](https://discuss.elastic.co/t/remove-blank-fields-from-output-file/75886 "2017-02-22T09:05:49Z")
I am pretty new to using log stash, so I need to know how to remove a line which does not match the grok pattern completely from the output. I have a 500 line input out of which 4-5 lines match my pattern. Presently, it … --- ## [Logstash jdbc driver error](https://discuss.elastic.co/t/logstash-jdbc-driver-error/76003)
**Author:** [@Navneet\_Mathpal](https://discuss.elastic.co/u/Navneet_Mathpal)\ **Replies:** 0\ **Last updated:** [February 22, 2017, 9:05am UTC](https://discuss.elastic.co/t/logstash-jdbc-driver-error/76003 "2017-02-22T09:05:45Z")
Hi , i am getting the below error while running the jdbc output in logstash 2.3. logstash is running fine for a week and then getting closed automatically :message=\>"JDBC - Exception repeated more than the maximum c… --- ## [Logstash output to be dumped to CASSANDRA](https://discuss.elastic.co/t/logstash-output-to-be-dumped-to-cassandra/76001)
**Author:** [@agrawalsaurabh](https://discuss.elastic.co/u/agrawalsaurabh)\ **Replies:** 0\ **Last updated:** [February 22, 2017, 8:54am UTC](https://discuss.elastic.co/t/logstash-output-to-be-dumped-to-cassandra/76001 "2017-02-22T08:54:19Z")
Hi, I have requirement to process apache access logs and get some meaningful analytics info. out of it. I ama newbie and installed Filebeat, logstash, elastic search and kibana. I can see logs processed and a snaps… --- ## [Logstash - send to different elasticsearch dependent on var](https://discuss.elastic.co/t/logstash-send-to-different-elasticsearch-dependent-on-var/75992)
**Author:** [@joshg](https://discuss.elastic.co/u/joshg)\ **Replies:** 1\ **Last updated:** [February 22, 2017, 8:39am UTC](https://discuss.elastic.co/t/logstash-send-to-different-elasticsearch-dependent-on-var/75992 "2017-02-22T08:39:59Z")
I have 2 elasticsearch environments. 1 for dev and 1 for prod. I want log stash to send to the correct environment dependent on an environment variable This is what my current output config looks like output { s… --- ## [Logstash -logs](https://discuss.elastic.co/t/logstash-logs/75996)
**Author:** [@Navneet\_Mathpal](https://discuss.elastic.co/u/Navneet_Mathpal)\ **Replies:** 0\ **Last updated:** [February 22, 2017, 7:53am UTC](https://discuss.elastic.co/t/logstash-logs/75996 "2017-02-22T07:53:38Z")
Hi , how we can check the logstash's logs , as my logstash is going down automatically once in a week . how to detect the root cause ? any help ? Thanks, Navneet [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1030) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1032)