# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=973 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 974 --- ## [KV filter not working](https://discuss.elastic.co/t/kv-filter-not-working/87019)
**Author:** [@munotshubham](https://discuss.elastic.co/u/munotshubham)\ **Replies:** 4\ **Last updated:** [May 31, 2017, 3:51pm UTC](https://discuss.elastic.co/t/kv-filter-not-working/87019 "2017-05-31T15:51:52Z")
config file: filter { if \[type\] == "syslog" { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:\\\[%{POSINT:syslog\_pid}\\\])?: %{GREEDYD… --- ## [Date filter not working for timestamps](https://discuss.elastic.co/t/date-filter-not-working-for-timestamps/87147)
**Author:** [@munotshubham](https://discuss.elastic.co/u/munotshubham)\ **Replies:** 2\ **Last updated:** [May 31, 2017, 3:49pm UTC](https://discuss.elastic.co/t/date-filter-not-working-for-timestamps/87147 "2017-05-31T15:49:49Z")
Logs seems to be fine, yet date filter is not working. If i remove the date filter, it takes the timestamp as string, so grok pattern is fine. Input format: 2017-04-03 11:40:59 2017-04-04 12:40:59 2017-04-05 13:40:5… --- ## [Logstash 1.5.0 grok regular expression . now includes also end of line character](https://discuss.elastic.co/t/logstash-1-5-0-grok-regular-expression-now-includes-also-end-of-line-character/970)
**Author:** [@Constantin\_Rata](https://discuss.elastic.co/u/Constantin_Rata)\ **Replies:** 1\ **Last updated:** [May 31, 2017, 3:46pm UTC](https://discuss.elastic.co/t/logstash-1-5-0-grok-regular-expression-now-includes-also-end-of-line-character/970 "2017-05-31T15:46:51Z")
Hi. I try to parse a log file that contains a stack trace. I use multiline coded in order to join multiple lines in one event. After I do this I want to use grok to identify the exception and store everything until end o… --- ## [Authenticating the Heroku Plugin](https://discuss.elastic.co/t/authenticating-the-heroku-plugin/87771)
**Author:** [@hobakill](https://discuss.elastic.co/u/hobakill)\ **Replies:** 0\ **Last updated:** [May 31, 2017, 2:51pm UTC](https://discuss.elastic.co/t/authenticating-the-heroku-plugin/87771 "2017-05-31T14:51:35Z")
I am relatively new to the ELK stack and I'm trying my best to figure out how to authenticate the Heroku input plugin. With a basic .yml I get: \[2017-05-31T14:42:25,853\]\[ERROR\]\[logstash.pipeline \] A plugin had… --- ## [Logstash input jms plugin problem for reading message from tibco ems broker](https://discuss.elastic.co/t/logstash-input-jms-plugin-problem-for-reading-message-from-tibco-ems-broker/43756)
**Author:** [@achutak](https://discuss.elastic.co/u/achutak)\ **Replies:** 1\ **Last updated:** [May 31, 2017, 2:49pm UTC](https://discuss.elastic.co/t/logstash-input-jms-plugin-problem-for-reading-message-from-tibco-ems-broker/43756 "2017-05-31T14:49:25Z")
in my elk setup with logstash version 2.2 .i am trying to pull the messages from tibco jms broker on certain queue for this my logstash config file is input{ jms { include\_header =\> false include\_properti… --- ## [Replacing @timestamp in logstash using grok fails \[Solved\]](https://discuss.elastic.co/t/replacing-timestamp-in-logstash-using-grok-fails-solved/87532)
**Author:** [@b.thoor](https://discuss.elastic.co/u/b.thoor)\ **Replies:** 6\ **Last updated:** [May 31, 2017, 2:03pm UTC](https://discuss.elastic.co/t/replacing-timestamp-in-logstash-using-grok-fails-solved/87532 "2017-05-31T14:03:07Z")
Hi. The setup is this filebeat (ws2012r2) 5.4.0 -\> logstash (rhel7) 5.4.0 -\> elasticsearch (rhel7) 5.4.0 and the presentation layer is kibana (rhel7) 5.4.0. This is a sample line from the logs: 2017-04-19 15:25:40,37… --- ## [Filebeat \> elasticsearch to filebeat \> logstash \> elasticsearch (syslog)](https://discuss.elastic.co/t/filebeat-elasticsearch-to-filebeat-logstash-elasticsearch-syslog/86671)
**Author:** [@Rgn](https://discuss.elastic.co/u/Rgn)\ **Replies:** 3\ **Last updated:** [May 31, 2017, 1:59pm UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-to-filebeat-logstash-elasticsearch-syslog/86671 "2017-05-31T13:59:43Z")
Hi there. i'm was using syslog module from filebeat to send data to elasticseach. In my kibana a can saw my data with all the additionnal field (ex: system.syslog.hostname, system.syslog.message system.syslog.pid, etc… --- ## [Multiple Mulitiline Patterns?](https://discuss.elastic.co/t/multiple-mulitiline-patterns/86995)
**Author:** [@Jaxon\_Kochel](https://discuss.elastic.co/u/Jaxon_Kochel)\ **Replies:** 2\ **Last updated:** [May 31, 2017, 1:40pm UTC](https://discuss.elastic.co/t/multiple-mulitiline-patterns/86995 "2017-05-31T13:40:09Z")
I'm wondering if there is any way to have multiple multiline patterns. I see this post https://discuss.elastic.co/t/solution-for-multiple-patterns-for-multiline-configuration/43807 which looks like he got something to … --- ## [Parsing thread number in grok](https://discuss.elastic.co/t/parsing-thread-number-in-grok/87743)
**Author:** [@zanza54](https://discuss.elastic.co/u/zanza54)\ **Replies:** 2\ **Last updated:** [May 31, 2017, 1:27pm UTC](https://discuss.elastic.co/t/parsing-thread-number-in-grok/87743 "2017-05-31T13:27:46Z")
Hi I'm trying to parse a logline in grok but I can't get past the thread number in my log lines. Example logline: 428 31/05/2017 08:11:53.243 T\[63\] INFO Service.Internal.ReturnUserType INTERNAL\\test\_cf --+End of… --- ## [Logstash(with input-jdbc) always sync ALL data into elasticsearch even the condition is false](https://discuss.elastic.co/t/logstash-with-input-jdbc-always-sync-all-data-into-elasticsearch-even-the-condition-is-false/85688)
**Author:** [@se681268](https://discuss.elastic.co/u/se681268)\ **Replies:** 3\ **Last updated:** [May 24, 2017, 8:43am UTC](https://discuss.elastic.co/t/logstash-with-input-jdbc-always-sync-all-data-into-elasticsearch-even-the-condition-is-false/85688 "2017-05-24T08:43:32Z")
Hi,everyone I had some trouble synchronizing data with logstash(with input-jdbc) between MySQL and elasticsearch.When data start sync,i can see it will keep rolling over and over again,even the condition is false. I made… --- ## [Grok match handel special characters, \_grokparsefailure](https://discuss.elastic.co/t/grok-match-handel-special-characters--grokparsefailure/55417)
**Author:** [@govereem](https://discuss.elastic.co/u/govereem)\ **Replies:** 3\ **Last updated:** [May 31, 2017, 12:14pm UTC](https://discuss.elastic.co/t/grok-match-handel-special-characters--grokparsefailure/55417 "2017-05-31T12:14:11Z")
I'm using this grok code grok { # check that fields match your IIS log settings match =\> \["message", "%{TIMESTAMP\_ISO8601:log\_timestamp} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring}… --- ## [Auto indent logstash conf files](https://discuss.elastic.co/t/auto-indent-logstash-conf-files/87719)
**Author:** [@Conall\_Prendergast](https://discuss.elastic.co/u/Conall_Prendergast)\ **Replies:** 0\ **Last updated:** [May 31, 2017, 10:47am UTC](https://discuss.elastic.co/t/auto-indent-logstash-conf-files/87719 "2017-05-31T10:47:03Z")
Is there a tool that anyone knows of that will auto-indent the logstash conf files? Preferably something that would work on embedded Ruby code. thanks --- ## [Logstash collectd config](https://discuss.elastic.co/t/logstash-collectd-config/87576)
**Author:** [@shrwn](https://discuss.elastic.co/u/shrwn)\ **Replies:** 7\ **Last updated:** [May 31, 2017, 10:16am UTC](https://discuss.elastic.co/t/logstash-collectd-config/87576 "2017-05-31T10:16:55Z")
Hi All, I have done logstash and collectd config as mentioned below: logstash.conf: output { redis { host =\> \["0.0.0.0", "0.0.0.1"\] data\_type =\> "list" key =\> "abc" password =\> "xyz" } } input { u… --- ## [Error while indexing data using elasticsearch output plugin](https://discuss.elastic.co/t/error-while-indexing-data-using-elasticsearch-output-plugin/87712)
**Author:** [@sumit\_monga](https://discuss.elastic.co/u/sumit_monga)\ **Replies:** 0\ **Last updated:** [May 31, 2017, 10:04am UTC](https://discuss.elastic.co/t/error-while-indexing-data-using-elasticsearch-output-plugin/87712 "2017-05-31T10:04:07Z")
I have a usecase where I have a big xml input file (currently testing with a 36 MB file but can be larger) and I am using logstash pipeline to convert this xml to json and then dumping the json to elasticsearch. I am ge… --- ## [JDBC plugin query taking too long to run](https://discuss.elastic.co/t/jdbc-plugin-query-taking-too-long-to-run/87638)
**Author:** [@asharma](https://discuss.elastic.co/u/asharma)\ **Replies:** 1\ **Last updated:** [May 31, 2017, 9:08am UTC](https://discuss.elastic.co/t/jdbc-plugin-query-taking-too-long-to-run/87638 "2017-05-31T09:08:11Z")
Hi, I am interested in how logstash handles scheduled jdbc inputs. If I schedule the jdbc input plugin to read some data every 5 mins but the query itself takes around 10 mins to run depending on server load. How will l… --- ## [Metric filter, based on logfile timestamp?](https://discuss.elastic.co/t/metric-filter-based-on-logfile-timestamp/87696)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 0\ **Last updated:** [May 31, 2017, 8:45am UTC](https://discuss.elastic.co/t/metric-filter-based-on-logfile-timestamp/87696 "2017-05-31T08:45:01Z")
Hi, I have a question about the metric filter. I played a bit with it some time ago, but I was not able to fit it to my needs. Maybe I just made some mistake. UseCase: I am having a log, with a timestamp. Timestamp… --- ## [Removing and renaming subsubfield with ruby code](https://discuss.elastic.co/t/removing-and-renaming-subsubfield-with-ruby-code/87695)
**Author:** [@toger](https://discuss.elastic.co/u/toger)\ **Replies:** 0\ **Last updated:** [May 31, 2017, 8:31am UTC](https://discuss.elastic.co/t/removing-and-renaming-subsubfield-with-ruby-code/87695 "2017-05-31T08:31:01Z")
Hi, I am trying to remove all the data\_type subsubfield from an JSON array that looks like : "doc": { "subdoc1": { "content": "valuex" "data\_type": "valuey" }, "subdoc2": { "c… --- ## [\[INFO \]\[logstash.codecs.nmap\] Using version 0.1.x codec plugin 'nmap'. This plugin isn't well supported by the community and likely has no maintainer](https://discuss.elastic.co/t/info-logstash-codecs-nmap-using-version-0-1-x-codec-plugin-nmap-this-plugin-isnt-well-supported-by-the-community-and-likely-has-no-maintainer/87596)
**Author:** [@Olha](https://discuss.elastic.co/u/Olha)\ **Replies:** 0\ **Last updated:** [May 30, 2017, 3:47pm UTC](https://discuss.elastic.co/t/info-logstash-codecs-nmap-using-version-0-1-x-codec-plugin-nmap-this-plugin-isnt-well-supported-by-the-community-and-likely-has-no-maintainer/87596 "2017-05-30T15:47:11Z")
Hello elastic-team, There is something odd with my Logstash when I am using logstash-codec-nmap. What I mean: 1. According to blog https://www.elastic.co/blog/using-nmap-logstash-to-gain-insight-into-your-network, I… --- ## [Logstash multiline filter not merging xml after new line](https://discuss.elastic.co/t/logstash-multiline-filter-not-merging-xml-after-new-line/57827)
**Author:** [@beelastic](https://discuss.elastic.co/u/beelastic)\ **Replies:** 14\ **Last updated:** [May 31, 2017, 7:54am UTC](https://discuss.elastic.co/t/logstash-multiline-filter-not-merging-xml-after-new-line/57827 "2017-05-31T07:54:41Z")
I want to parse xml files with logstash Example File: \ \\\\SPI\\\\ **Author:** [@ashokchippa](https://discuss.elastic.co/u/ashokchippa)\ **Replies:** 0\ **Last updated:** [May 31, 2017, 5:24am UTC](https://discuss.elastic.co/t/get-data-from-solr-collection-to-hadoop/87668 "2017-05-31T05:24:46Z") Hi , We have not seen any input solr documents. We want to get the data from solr using logstash. Can anyone help on how to get the data from solr collection and put that data into hadoop. Thanks, Ashok Ch --- ## [Elastic runner exception on logstash startup](https://discuss.elastic.co/t/elastic-runner-exception-on-logstash-startup/87604)
**Author:** [@anil\_kumar4](https://discuss.elastic.co/u/anil_kumar4)\ **Replies:** 1\ **Last updated:** [May 31, 2017, 5:02am UTC](https://discuss.elastic.co/t/elastic-runner-exception-on-logstash-startup/87604 "2017-05-31T05:02:47Z")
Hi Team, I installed/configured Elastic, logstash, kabana on individual nodes. I am seeing the below issue while logstash server startup, i followed this doc for logstash installations Installing Logstash (Optional) |… --- ## [Finding same pattern multiple times in one log file](https://discuss.elastic.co/t/finding-same-pattern-multiple-times-in-one-log-file/56196)
**Author:** [@DJ1995](https://discuss.elastic.co/u/DJ1995)\ **Replies:** 6\ **Last updated:** [May 30, 2017, 11:30pm UTC](https://discuss.elastic.co/t/finding-same-pattern-multiple-times-in-one-log-file/56196 "2017-05-30T23:30:15Z")
Hi, I'm new to using the elk so the question may be easy / make no sense. One pattern appears 3 times with different numbers inside my log file. I can use the grok filter with the match config to find the pattern at all … --- ## [Kafka Output Plugin Appears to do Nothing](https://discuss.elastic.co/t/kafka-output-plugin-appears-to-do-nothing/87650)
**Author:** [@Claxor](https://discuss.elastic.co/u/Claxor)\ **Replies:** 0\ **Last updated:** [May 30, 2017, 11:26pm UTC](https://discuss.elastic.co/t/kafka-output-plugin-appears-to-do-nothing/87650 "2017-05-30T23:26:50Z")
Hello you wonderful people. I'm trying to use the Kafka output plugin and thus far I so no evidence it is being invoked. The debug shows the kafka output-plugin variables on logstash startup, then nothing until logst… --- ## [Rollover issue using logstash](https://discuss.elastic.co/t/rollover-issue-using-logstash/86892)
**Author:** [@kamran.matloob](https://discuss.elastic.co/u/kamran.matloob)\ **Replies:** 2\ **Last updated:** [May 30, 2017, 10:08pm UTC](https://discuss.elastic.co/t/rollover-issue-using-logstash/86892 "2017-05-30T22:08:38Z")
Hi, I am using 5.1 elk stack. My task is to rollover 2 indices. I came to know through below link that we can rollover through curator https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/ex\_rollover.html Bu… --- ## [Copy the value of a field into an event and paste it into another field and another event](https://discuss.elastic.co/t/copy-the-value-of-a-field-into-an-event-and-paste-it-into-another-field-and-another-event/87639)
**Author:** [@rocavalcante](https://discuss.elastic.co/u/rocavalcante)\ **Replies:** 0\ **Last updated:** [May 30, 2017, 9:42pm UTC](https://discuss.elastic.co/t/copy-the-value-of-a-field-into-an-event-and-paste-it-into-another-field-and-another-event/87639 "2017-05-30T21:42:32Z")
Hi Everyone, My doubt is for example: { "offset" =\> 7, "word1" =\> "XPTO", \<- Copy this value "@metadata" =\> { "beat" =\> "filebeat", "type" =\> "log" }, "input\_type" =\> "… --- ## [Unit Testing Grok Patterns / Logstash Configs](https://discuss.elastic.co/t/unit-testing-grok-patterns-logstash-configs/87605)
**Author:** [@hahe](https://discuss.elastic.co/u/hahe)\ **Replies:** 2\ **Last updated:** [May 30, 2017, 9:28pm UTC](https://discuss.elastic.co/t/unit-testing-grok-patterns-logstash-configs/87605 "2017-05-30T21:28:30Z")
I'm wondering how others are managing evolving Grok patterns / logstash configurations. Are there any best practices / approaches to make sure that after making changes in a logstash configuration messages that have b… --- ## [Copy field on event](https://discuss.elastic.co/t/copy-field-on-event/87631)
**Author:** [@rocavalcante](https://discuss.elastic.co/u/rocavalcante)\ **Replies:** 0\ **Last updated:** [May 30, 2017, 8:22pm UTC](https://discuss.elastic.co/t/copy-field-on-event/87631 "2017-05-30T20:22:47Z")
I'm new to logstash, I'm trying to use logstash and filebeat. 'The filebeat sends the events (lines) of a log file \* .txt, and logstash does the filter and cature the messages I want, however the main messages are in the… --- ## [Parsing very large xml file](https://discuss.elastic.co/t/parsing-very-large-xml-file/87608)
**Author:** [@timj123](https://discuss.elastic.co/u/timj123)\ **Replies:** 0\ **Last updated:** [May 30, 2017, 5:50pm UTC](https://discuss.elastic.co/t/parsing-very-large-xml-file/87608 "2017-05-30T17:50:28Z")
I have a very large xml file around 1.9M lines of xml that I would like to do the following: Use two TAGs under the root of the file to create a timestamp that ALL events in step two will use. Create an events using t… --- ## [Field value should not be updated if document already exists](https://discuss.elastic.co/t/field-value-should-not-be-updated-if-document-already-exists/87349)
**Author:** [@skairam](https://discuss.elastic.co/u/skairam)\ **Replies:** 1\ **Last updated:** [May 30, 2017, 5:59pm UTC](https://discuss.elastic.co/t/field-value-should-not-be-updated-if-document-already-exists/87349 "2017-05-30T17:59:12Z")
HI, I have fields CRTE\_TMS, UPDATE\_TMS. In my logstash configuration, i have the following mutate, which adds fields with current time stamps value and i have doc\_as\_upsert =\> true. This script is updating the exist… --- ## [Multiline between tags](https://discuss.elastic.co/t/multiline-between-tags/87607)
**Author:** [@timj123](https://discuss.elastic.co/u/timj123)\ **Replies:** 0\ **Last updated:** [May 30, 2017, 5:22pm UTC](https://discuss.elastic.co/t/multiline-between-tags/87607 "2017-05-30T17:22:12Z")
I'm struggling with a vary large xml file that I need multiple multi-line messages. Is it possible to combine two xml fields to create a timestamp that all messages in the current file will use to create their event. … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=972) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=974)