# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=669 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 670 --- ## [Grok parse failures .. Grok syntax](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083)
**Author:** [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Replies:** 21\ **Last updated:** [October 12, 2018, 8:21am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083 "2018-10-12T08:21:44Z")
Hi, I have matching records for the below grok pattern \[%{TIMESTAMP\_ISO8601:timestamp}\] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} \[%{NUMBER:ID}:%{GREEDYDATA:STATE}\] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abort… --- ## [NetFlow v9 Interface/VRF mappings](https://discuss.elastic.co/t/netflow-v9-interface-vrf-mappings/152186)
**Author:** [@1fe60245fe160bc0f8ab](https://discuss.elastic.co/u/1fe60245fe160bc0f8ab)\ **Replies:** 0\ **Last updated:** [October 12, 2018, 7:59am UTC](https://discuss.elastic.co/t/netflow-v9-interface-vrf-mappings/152186 "2018-10-12T07:59:16Z")
Hi all, We're currently using Logstash's NetFlow plugin in order to decode and process all sorts of NetFlow streams (v5, v9, IPFIX) coming from various sources (Checkpoint, Cisco, VMware and others). We've encountered … --- ## [Logstash 6.4.0 problem](https://discuss.elastic.co/t/logstash-6-4-0-problem/151853)
**Author:** [@magunes](https://discuss.elastic.co/u/magunes)\ **Replies:** 3\ **Last updated:** [October 12, 2018, 7:51am UTC](https://discuss.elastic.co/t/logstash-6-4-0-problem/151853 "2018-10-12T07:51:58Z")
Hello, I have just installed logstash 6.4.0 and getting error below. I am using exactly the same config file (which I have used with 6.1.3 so far) Is this a bug or something changed with the new version? \[2018-10-10T… --- ## [Elapsed\_expired\_error](https://discuss.elastic.co/t/elapsed-expired-error/152184)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 0\ **Last updated:** [October 12, 2018, 7:44am UTC](https://discuss.elastic.co/t/elapsed-expired-error/152184 "2018-10-12T07:44:26Z")
hi elk community, when elapsed\_expired\_error event generates from logstash, is that possible to trace the value of end\_tag? plz point out to me. i appreciate that!!! --- ## [Logstash Newby](https://discuss.elastic.co/t/logstash-newby/152181)
**Author:** [@shigatsu](https://discuss.elastic.co/u/shigatsu)\ **Replies:** 0\ **Last updated:** [October 12, 2018, 7:41am UTC](https://discuss.elastic.co/t/logstash-newby/152181 "2018-10-12T07:41:42Z")
Using filebeat and logstash, i'm having trouble with my grok pattern. here's may sample.conf input { beats { port =\> "5044" } } filter { grok { match =\> { "message" =\> "%{DATE:logdate}%{SPACE}%{NUMBER:collected\_… --- ## [File input in read mode not deleting the file after parsing](https://discuss.elastic.co/t/file-input-in-read-mode-not-deleting-the-file-after-parsing/152053)
**Author:** [@rakeshpatri](https://discuss.elastic.co/u/rakeshpatri)\ **Replies:** 1\ **Last updated:** [October 12, 2018, 6:33am UTC](https://discuss.elastic.co/t/file-input-in-read-mode-not-deleting-the-file-after-parsing/152053 "2018-10-12T06:33:03Z")
Version: logstash-input-file (4.1.6) Operating System: logstash 6.4.1 image (container) Config File: input { file { id =\> "client-files" mode =\> "read" path =\> \[ "/usr/share/logstash/plugins/message/\*.gz"\] add\_fie… --- ## [Cisco Meraki Syslog - Using Two Grok Patterns in the same filter](https://discuss.elastic.co/t/cisco-meraki-syslog-using-two-grok-patterns-in-the-same-filter/152105)
**Author:** [@dts](https://discuss.elastic.co/u/dts)\ **Replies:** 1\ **Last updated:** [October 12, 2018, 4:05am UTC](https://discuss.elastic.co/t/cisco-meraki-syslog-using-two-grok-patterns-in-the-same-filter/152105 "2018-10-12T04:05:38Z")
Hello, I have created a conf with two grok patterns. When I run the conf it seems to only process the first pattern and not the second one. I have tested both pattens in a couple of online Grok sites and they both work.… --- ## [Using RegEx for multiple matches to store in single field or dynamic fields](https://discuss.elastic.co/t/using-regex-for-multiple-matches-to-store-in-single-field-or-dynamic-fields/151742)
**Author:** [@nicpenning](https://discuss.elastic.co/u/nicpenning)\ **Replies:** 0\ **Last updated:** [October 10, 2018, 3:50am UTC](https://discuss.elastic.co/t/using-regex-for-multiple-matches-to-store-in-single-field-or-dynamic-fields/151742 "2018-10-10T03:50:45Z")
Is it possible to parse an event field for multiple values and have all matches be stored in multiple new fields dynamically? For example, if \[message\] contains multiple hits on a regex searching for URLs, could they al… --- ## [Logstash errors while executing ruby code (Ruby exception occurred: allocator undefined for Floa)](https://discuss.elastic.co/t/logstash-errors-while-executing-ruby-code-ruby-exception-occurred-allocator-undefined-for-floa/151939)
**Author:** [@Dawn\_Okem](https://discuss.elastic.co/u/Dawn_Okem)\ **Replies:** 1\ **Last updated:** [October 11, 2018, 11:07pm UTC](https://discuss.elastic.co/t/logstash-errors-while-executing-ruby-code-ruby-exception-occurred-allocator-undefined-for-floa/151939 "2018-10-11T23:07:26Z")
Was trying to execute below code. but logstash keeps spitting an error out. if \[received\_at\] and \[sent\_at\] { ruby { init =\> "require 'time'" code =\> " received\_by\_finacle = (Time.parse(event.get('received\_at').… --- ## [How to calculate timediff (in milliseconds) of two fields in the same event](https://discuss.elastic.co/t/how-to-calculate-timediff-in-milliseconds-of-two-fields-in-the-same-event/151711)
**Author:** [@Dawn\_Okem](https://discuss.elastic.co/u/Dawn_Okem)\ **Replies:** 3\ **Last updated:** [October 11, 2018, 11:00pm UTC](https://discuss.elastic.co/t/how-to-calculate-timediff-in-milliseconds-of-two-fields-in-the-same-event/151711 "2018-10-11T23:00:51Z")
My question is similar to the one asked here but i want to be able to get the time difference in milliseconds. I saw a ruby solution here, but how do i modify the code to give me the difference in milliseconds instead … --- ## [Logstash.yml Elastic Cloud Settings](https://discuss.elastic.co/t/logstash-yml-elastic-cloud-settings/152116)
**Author:** [@schoffelman](https://discuss.elastic.co/u/schoffelman)\ **Replies:** 1\ **Last updated:** [October 11, 2018, 7:25pm UTC](https://discuss.elastic.co/t/logstash-yml-elastic-cloud-settings/152116 "2018-10-11T19:25:52Z")
In my Logstash.yml file, I've added both my cloud.id and cloud.auth information (both with and without quotes for the value). In each case, when running my conf it tries to connect to my 127.0.0.1 localhost. I was able t… --- ## [Dealing with repeating XML fields](https://discuss.elastic.co/t/dealing-with-repeating-xml-fields/152114)
**Author:** [@rcoundon](https://discuss.elastic.co/u/rcoundon)\ **Replies:** 0\ **Last updated:** [October 11, 2018, 7:15pm UTC](https://discuss.elastic.co/t/dealing-with-repeating-xml-fields/152114 "2018-10-11T19:15:15Z")
Hi I have XML in the following format: \ \ \2018-10-10\ \2296\ \1234\ \ **Author:** [@venkat\_venkat](https://discuss.elastic.co/u/venkat_venkat)\ **Replies:** 0\ **Last updated:** [October 11, 2018, 6:26pm UTC](https://discuss.elastic.co/t/how-to-add-field-value-to-id-value-using-logstash/152107 "2018-10-11T18:26:43Z") I have a requirement that has to add one of the field value in the documet to \_id value using logstash while indexing. Ex: { This the document coming to Elasticsearch using logstash. \_id : 1 “coloum\_XXXX": 10000032… --- ## [Logstash with rotating logs](https://discuss.elastic.co/t/logstash-with-rotating-logs/152031)
**Author:** [@serozhka](https://discuss.elastic.co/u/serozhka)\ **Replies:** 1\ **Last updated:** [October 11, 2018, 3:55pm UTC](https://discuss.elastic.co/t/logstash-with-rotating-logs/152031 "2018-10-11T15:55:50Z")
Hi! We use logstash 6.2.4. which sends logs to Kafka nodes. We need to to track application logs which are rotated each 24h, it is not possible to control log file names and their format is like: mylogfile\_20181007\_000… --- ## [Need Logstash logs like filebeat](https://discuss.elastic.co/t/need-logstash-logs-like-filebeat/150729)
**Author:** [@Nithin\_Venugopal](https://discuss.elastic.co/u/Nithin_Venugopal)\ **Replies:** 3\ **Last updated:** [October 11, 2018, 3:43pm UTC](https://discuss.elastic.co/t/need-logstash-logs-like-filebeat/150729 "2018-10-11T15:43:13Z")
Is there any way to get logstash to log how filebeat does as below {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":9}},"total":{"ticks":10,"time":{"ms":23},"value":10},"user":{"ticks":10,"tim… --- ## [Error with msgpack input codec : TypeError: wrong argument type Fixnum (expected Hash)](https://discuss.elastic.co/t/error-with-msgpack-input-codec-typeerror-wrong-argument-type-fixnum-expected-hash/152013)
**Author:** [@tomahim](https://discuss.elastic.co/u/tomahim)\ **Replies:** 0\ **Last updated:** [October 11, 2018, 8:42am UTC](https://discuss.elastic.co/t/error-with-msgpack-input-codec-typeerror-wrong-argument-type-fixnum-expected-hash/152013 "2018-10-11T08:42:16Z")
Hello, I am trying to decode msgpack serialized events from a Kafka instance, by using the msgpack input codec. This the configuration file i'm using : input { kafka { bootstrap\_servers =\> "${KAFKA\_BOOTSTRAP\_SER… --- ## [Csv output separator](https://discuss.elastic.co/t/csv-output-separator/151876)
**Author:** [@ioannisnikolakis](https://discuss.elastic.co/u/ioannisnikolakis)\ **Replies:** 2\ **Last updated:** [October 11, 2018, 2:48pm UTC](https://discuss.elastic.co/t/csv-output-separator/151876 "2018-10-11T14:48:24Z")
Hi, I need to change the separator of my output file to semicolon instead of a comma, due to text fields, containing commas, being present. Is this feasible ? Thank you in advance, --- ## [When in,filter,out not the same event](https://discuss.elastic.co/t/when-in-filter-out-not-the-same-event/151988)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 1\ **Last updated:** [October 11, 2018, 2:20pm UTC](https://discuss.elastic.co/t/when-in-filter-out-not-the-same-event/151988 "2018-10-11T14:20:01Z")
hi team, i add a couple of elapsed filters in my logstash pipeline it works properly. but i'm wonder why in,filter and out events not same eg: when i start logstash service, in, filter and output events show same a … --- ## [Character encoding problems Filebeat & Logstash](https://discuss.elastic.co/t/character-encoding-problems-filebeat-logstash/150353)
**Author:** [@keysersozee](https://discuss.elastic.co/u/keysersozee)\ **Replies:** 2\ **Last updated:** [October 11, 2018, 1:58pm UTC](https://discuss.elastic.co/t/character-encoding-problems-filebeat-logstash/150353 "2018-10-11T13:58:05Z")
Hello, I've read many topics in about this problem but it's still not resolved for me. I have a fileabeat 6.2.2 installed on Windows, it will send logs logs to a Logstash 6.2.2 installed on Linux. This my filebeat con… --- ## [Logstash grok / dissect filter over array](https://discuss.elastic.co/t/logstash-grok-dissect-filter-over-array/151992)
**Author:** [@ftds](https://discuss.elastic.co/u/ftds)\ **Replies:** 4\ **Last updated:** [October 11, 2018, 1:57pm UTC](https://discuss.elastic.co/t/logstash-grok-dissect-filter-over-array/151992 "2018-10-11T13:57:41Z")
Hello, we have a log line that we want to parse with Logstash that looks like this: from=\ to=\ to=\ to=\ To parse this line, we use the Logsta… --- ## [Logstash failed to send logs to elasticsearch](https://discuss.elastic.co/t/logstash-failed-to-send-logs-to-elasticsearch/151039)
**Author:** [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)\ **Replies:** 8\ **Last updated:** [October 11, 2018, 11:54am UTC](https://discuss.elastic.co/t/logstash-failed-to-send-logs-to-elasticsearch/151039 "2018-10-11T11:54:02Z")
Hello, I am new to ELK and I just installed ELK Stack 6.4.2. I am trying to send windows Events to ELK. I installed Elasticsearch and Kibana and then Logstash. I installed winlogbeat in a windows server and configured … --- ## [JDBC to ElasticSearch: duplicate records?](https://discuss.elastic.co/t/jdbc-to-elasticsearch-duplicate-records/150153)
**Author:** [@jgabor](https://discuss.elastic.co/u/jgabor)\ **Replies:** 1\ **Last updated:** [October 11, 2018, 11:49am UTC](https://discuss.elastic.co/t/jdbc-to-elasticsearch-duplicate-records/150153 "2018-10-11T11:49:40Z")
Hi! I'm working on a PoC where we are evaluating how to ingest data from an Oracle table into ElasticSearch using Logstash and multiple connections to the Oracle DB. While doing this I observed a strange bahviour I can… --- ## [\ **Author:** [@Billz1026](https://discuss.elastic.co/u/Billz1026)\ **Replies:** 3\ **Last updated:** [October 11, 2018, 11:42am UTC](https://discuss.elastic.co/t/regexperror-end-pattern-with-unmatched-parenthesis/152036 "2018-10-11T11:42:58Z") HI, First of all I have to say I am a newbie in the ELK domain. I have written a simple config file but is giving an error each time i try to run logstash. Following is the config file. \`input{ file { path =\> "/var/… --- ## [Filltering messages by tags problem with some messages](https://discuss.elastic.co/t/filltering-messages-by-tags-problem-with-some-messages/151856)
**Author:** [@Pietia](https://discuss.elastic.co/u/Pietia)\ **Replies:** 1\ **Last updated:** [October 11, 2018, 11:11am UTC](https://discuss.elastic.co/t/filltering-messages-by-tags-problem-with-some-messages/151856 "2018-10-11T11:11:37Z")
So I have a problem with logstash: some messages are being sent to default index:(below is the message that went into 'default' index. { "\_index": "default-logs-2018.10.10", "\_type": "doc", "\_id": "MioyXmYBMkEJh… --- ## [Getting length of array in Logstash using ruby filter](https://discuss.elastic.co/t/getting-length-of-array-in-logstash-using-ruby-filter/151999)
**Author:** [@dx123](https://discuss.elastic.co/u/dx123)\ **Replies:** 1\ **Last updated:** [October 11, 2018, 10:20am UTC](https://discuss.elastic.co/t/getting-length-of-array-in-logstash-using-ruby-filter/151999 "2018-10-11T10:20:46Z")
I have a field, Ports, that is in arrays and is nested. Examples of my data "ports": \["TCP1", "TCP2", 'TCP3"\] "ports": \["UDP1", "UDP2"\] However each of these feeds have different number of elements in them. I have to… --- ## [Ruby get element of an array](https://discuss.elastic.co/t/ruby-get-element-of-an-array/152004)
**Author:** [@Pierre2](https://discuss.elastic.co/u/Pierre2)\ **Replies:** 1\ **Last updated:** [October 11, 2018, 10:12am UTC](https://discuss.elastic.co/t/ruby-get-element-of-an-array/152004 "2018-10-11T10:12:13Z")
Hello, I want to get the element of an array called 'Choix menu SVI' at the position "i" and check if it is equal to '9'. I tried this bit it does not work. How can I do that ? if(event.get('Choix menu SVI'\[i\]))=='9' … --- ## [Logstash random fail to open TCP connection](https://discuss.elastic.co/t/logstash-random-fail-to-open-tcp-connection/152024)
**Author:** [@Fernando\_Acc](https://discuss.elastic.co/u/Fernando_Acc)\ **Replies:** 0\ **Last updated:** [October 11, 2018, 9:41am UTC](https://discuss.elastic.co/t/logstash-random-fail-to-open-tcp-connection/152024 "2018-10-11T09:41:56Z")
I'm getting a random error while running logstash. It seems like it happens even with a very low number of events. Our development includes custom filters that run querys and updates on Elasticsearch (Elastic Cloud) \[… --- ## [Problem to send information to Icinga](https://discuss.elastic.co/t/problem-to-send-information-to-icinga/151412)
**Author:** [@soportesms](https://discuss.elastic.co/u/soportesms)\ **Replies:** 1\ **Last updated:** [October 11, 2018, 8:18am UTC](https://discuss.elastic.co/t/problem-to-send-information-to-icinga/151412 "2018-10-11T08:18:41Z")
Hello, I have a problem to send events logstast-Icinga. My output: output { icinga { host =\> "X.X.X.X" user =\> "logstash" password =\> "xxxxxxx" ssl\_verify =\> "false" action … --- ## [Logstash-imap-input-error](https://discuss.elastic.co/t/logstash-imap-input-error/151839)
**Author:** [@souf\_el\_badraoui](https://discuss.elastic.co/u/souf_el_badraoui)\ **Replies:** 2\ **Last updated:** [October 11, 2018, 7:38am UTC](https://discuss.elastic.co/t/logstash-imap-input-error/151839 "2018-10-11T07:38:30Z")
Hi, We use logstash 5.6.5 Following the example here: https://qbox.io/blog/indexing-emails-to-elasticsearch-logstash-imap , I got this error: \[2018-10-10T13:54:43,586\]\[ERROR\]\[logstash.pipeline \] A plugin had an… --- ## [Want to stitch 1 DHCP log line to several NAT log line matching with MAC address](https://discuss.elastic.co/t/want-to-stitch-1-dhcp-log-line-to-several-nat-log-line-matching-with-mac-address/151843)
**Author:** [@A\_S\_M\_Shamim\_Reza](https://discuss.elastic.co/u/A_S_M_Shamim_Reza)\ **Replies:** 1\ **Last updated:** [October 11, 2018, 7:20am UTC](https://discuss.elastic.co/t/want-to-stitch-1-dhcp-log-line-to-several-nat-log-line-matching-with-mac-address/151843 "2018-10-11T07:20:04Z")
I am taking DHCP assign log and NAT log from mikrotik router. I have manage to split those logs with specific terms. Now i want to stitch the DHCP assign log with the NAT log, where the MAC address and IP address will b… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=668) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=670)