# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=661 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 662 --- ## [Using ELK, I'm unsure how to use right filter in logstash config file for pairs](https://discuss.elastic.co/t/using-elk-im-unsure-how-to-use-right-filter-in-logstash-config-file-for-pairs/153658)
**Author:** [@Mark\_McGown](https://discuss.elastic.co/u/Mark_McGown)\ **Replies:** 1\ **Last updated:** [October 24, 2018, 8:04pm UTC](https://discuss.elastic.co/t/using-elk-im-unsure-how-to-use-right-filter-in-logstash-config-file-for-pairs/153658 "2018-10-24T20:04:37Z")
I have a dollarVars term filled with things in this format... Message 1: Header1|Value1|Header2|Value2|Header3|Value3| Message 2: Header2|Value2|Header4|Value4|Header5|Value5| Message 3: Header3|Value3|Header4|Valu… --- ## [Trouble parsing xml file](https://discuss.elastic.co/t/trouble-parsing-xml-file/153582)
**Author:** [@Miguel\_Leite](https://discuss.elastic.co/u/Miguel_Leite)\ **Replies:** 2\ **Last updated:** [October 24, 2018, 7:59pm UTC](https://discuss.elastic.co/t/trouble-parsing-xml-file/153582 "2018-10-24T19:59:20Z")
Hey there, New to ELK and I was trying to parse the following xml file: \ **Author:** [@Callahan](https://discuss.elastic.co/u/Callahan)\ **Replies:** 0\ **Last updated:** [October 24, 2018, 6:59pm UTC](https://discuss.elastic.co/t/xml-filter-or-prune-filter-more-suited-for-this-log-file-or-simply-grok-it/153862 "2018-10-24T18:59:29Z") This is the output I get from querying a Sonos speaker using a Python script that I've got outputting to a file. I have 2 questions. Firstly, my aim is to extract only a small segment of info from this text file. Specifi… --- ## [Grok Pattern file iteration](https://discuss.elastic.co/t/grok-pattern-file-iteration/153847)
**Author:** [@camay123](https://discuss.elastic.co/u/camay123)\ **Replies:** 0\ **Last updated:** [October 24, 2018, 4:19pm UTC](https://discuss.elastic.co/t/grok-pattern-file-iteration/153847 "2018-10-24T16:19:41Z")
I got a feasability question, I am using logstash to parse docs, on a specific field of those docs, I want to match against a set of rules defined in a grok patterns file. MY file would look like: RULE1 (regex1) RULE… --- ## [Building ETL form ES to csv, last run date needed](https://discuss.elastic.co/t/building-etl-form-es-to-csv-last-run-date-needed/153841)
**Author:** [@Martin\_1982](https://discuss.elastic.co/u/Martin_1982)\ **Replies:** 0\ **Last updated:** [October 24, 2018, 3:32pm UTC](https://discuss.elastic.co/t/building-etl-form-es-to-csv-last-run-date-needed/153841 "2018-10-24T15:32:12Z")
Hello, im using logstash to extract data from ES to store it in csv (finally it should end up in a s3 bucket, but the output part is not so important here). I set up the following configuration: input { elasticsearch … --- ## [Logstash 6.4.2 Netflow Failure](https://discuss.elastic.co/t/logstash-6-4-2-netflow-failure/153837)
**Author:** [@mctsdmr](https://discuss.elastic.co/u/mctsdmr)\ **Replies:** 0\ **Last updated:** [October 24, 2018, 3:12pm UTC](https://discuss.elastic.co/t/logstash-6-4-2-netflow-failure/153837 "2018-10-24T15:12:03Z")
Hello folks, I want to collect the netflow data provided by my network using elastic stack(elasticsearch-kibana-logstash/netflowModule). To get logstash netflow module up and running, I use the the following command: … --- ## [Help with grok patterns for custom Postfix logs](https://discuss.elastic.co/t/help-with-grok-patterns-for-custom-postfix-logs/153835)
**Author:** [@decibel83](https://discuss.elastic.co/u/decibel83)\ **Replies:** 0\ **Last updated:** [October 24, 2018, 2:57pm UTC](https://discuss.elastic.co/t/help-with-grok-patterns-for-custom-postfix-logs/153835 "2018-10-24T14:57:14Z")
Hi, I am parsing Postfix logs using the good work from https://github.com/topmedia/logstash-postfix. Now I'm trying to customise the grok patterns because I have to parse some custom logs created with the Postfix heade… --- ## [Geoip not apeearing at indexes](https://discuss.elastic.co/t/geoip-not-apeearing-at-indexes/153815)
**Author:** [@paulo\_bruck](https://discuss.elastic.co/u/paulo_bruck)\ **Replies:** 2\ **Last updated:** [October 24, 2018, 2:14pm UTC](https://discuss.elastic.co/t/geoip-not-apeearing-at-indexes/153815 "2018-10-24T14:14:13Z")
Hy all I have read a lot of things about this subject, but none of them solve my problem. Using kibana,elasticsearch and logstash v6.4.2-1 Iniciating logstash appears this: \[2018-10-24T10:00:15,816\]\[INFO \]\[logstash.o… --- ## [Https://discuss.elastic.co/t/grok-will-not-change-field-type-to-int-or-ip/63756/4](https://discuss.elastic.co/t/https-discuss-elastic-co-t-grok-will-not-change-field-type-to-int-or-ip-63756-4/153793)
**Author:** [@Ilia](https://discuss.elastic.co/u/Ilia)\ **Replies:** 2\ **Last updated:** [October 24, 2018, 2:03pm UTC](https://discuss.elastic.co/t/https-discuss-elastic-co-t-grok-will-not-change-field-type-to-int-or-ip-63756-4/153793 "2018-10-24T14:03:19Z")
Continuing the discussion from Grok will not change field type to Int or IP: I'm using elasticsearch-6.4.2 along with logstash-6.4.2 In the /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsea… --- ## [Help with grok pattern for logs parsen in logstash](https://discuss.elastic.co/t/help-with-grok-pattern-for-logs-parsen-in-logstash/153819)
**Author:** [@bab](https://discuss.elastic.co/u/bab)\ **Replies:** 0\ **Last updated:** [October 24, 2018, 1:48pm UTC](https://discuss.elastic.co/t/help-with-grok-pattern-for-logs-parsen-in-logstash/153819 "2018-10-24T13:48:57Z")
Hey all, i have a problem with my grok pattern i have 2 type of logs with have one different and if i create two grok matches with the patterns (all similar until LOGLEVEL pattern) look below: my logs \[9/16/18 19:58:… --- ## [How to handle multi line logs in logstash](https://discuss.elastic.co/t/how-to-handle-multi-line-logs-in-logstash/153609)
**Author:** [@Parvatayya\_Malimath](https://discuss.elastic.co/u/Parvatayya_Malimath)\ **Replies:** 4\ **Last updated:** [October 24, 2018, 1:44pm UTC](https://discuss.elastic.co/t/how-to-handle-multi-line-logs-in-logstash/153609 "2018-10-24T13:44:16Z")
Hi I have such logs, i am not sure how handle this multi line logs, as you can see. its not consistent, we have multi lines at times and single lines at times. please help, how to parse them 2018-10-22 15:32:24.407 DE… --- ## [JDBC Input Plugin Count I/O Error](https://discuss.elastic.co/t/jdbc-input-plugin-count-i-o-error/153652)
**Author:** [@Johnathan](https://discuss.elastic.co/u/Johnathan)\ **Replies:** 1\ **Last updated:** [October 24, 2018, 1:40pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-count-i-o-error/153652 "2018-10-24T13:40:03Z")
I have a rather large query that will be picking up several million records from a Postgres DB and indexing into Elasticsearch. I have always used the jdbc input plugin for taking data from a RDBMS and putting it into El… --- ## [VMware/vCenter logging](https://discuss.elastic.co/t/vmware-vcenter-logging/153797)
**Author:** [@Ranganathc](https://discuss.elastic.co/u/Ranganathc)\ **Replies:** 0\ **Last updated:** [October 24, 2018, 12:36pm UTC](https://discuss.elastic.co/t/vmware-vcenter-logging/153797 "2018-10-24T12:36:58Z")
Hi All, I want to get logs from my VMware hosts and VMs into logstash. Does anyone has procedure document to forward the logs to ELK stack? --- ## [How to loop over an array in logstash](https://discuss.elastic.co/t/how-to-loop-over-an-array-in-logstash/153781)
**Author:** [@moshikoy](https://discuss.elastic.co/u/moshikoy)\ **Replies:** 0\ **Last updated:** [October 24, 2018, 11:21am UTC](https://discuss.elastic.co/t/how-to-loop-over-an-array-in-logstash/153781 "2018-10-24T11:21:45Z")
Hi, I was wondering what is the correct syntax for what i'm trying to achieve. I'm splitting a field using a separator, the value may be splitted to 2 or more sub-fields. I'd like to put these sub-fields in new fields… --- ## [Handle null values in json](https://discuss.elastic.co/t/handle-null-values-in-json/153774)
**Author:** [@prem\_kumar](https://discuss.elastic.co/u/prem_kumar)\ **Replies:** 0\ **Last updated:** [October 24, 2018, 11:03am UTC](https://discuss.elastic.co/t/handle-null-values-in-json/153774 "2018-10-24T11:03:19Z")
I have a JSON line this { "name": "prem", "level": null } My Logstash configuration looks like this filter { grok { match =\> {{TIMESTAMP\_ISO8601:timestamp} %{GREEDYDATA:logD}" } } json{ source =\> "logD" target … --- ## [Unable to load xml file in logstash](https://discuss.elastic.co/t/unable-to-load-xml-file-in-logstash/151318)
**Author:** [@balumurari1](https://discuss.elastic.co/u/balumurari1)\ **Replies:** 4\ **Last updated:** [October 24, 2018, 10:21am UTC](https://discuss.elastic.co/t/unable-to-load-xml-file-in-logstash/151318 "2018-10-24T10:21:17Z")
Hello, can you please provide a sample example to upload xml file using logstash. I have tried a lot but unable to upload an xml file in logstash in windows machine. Waiting for the response. Ex: My XML file is as foll… --- ## [Too many dynamic script compilations with inline script](https://discuss.elastic.co/t/too-many-dynamic-script-compilations-with-inline-script/153597)
**Author:** [@Matthew\_Field](https://discuss.elastic.co/u/Matthew_Field)\ **Replies:** 1\ **Last updated:** [October 24, 2018, 10:08am UTC](https://discuss.elastic.co/t/too-many-dynamic-script-compilations-with-inline-script/153597 "2018-10-24T10:08:30Z")
I am using an inline script with logstash. However i am getting "too many dynamic script compilations" error. I understand that this should not happen with an inline script, but it seems to be happening even if i simpl… --- ## [Grok Filter from NAS Synology](https://discuss.elastic.co/t/grok-filter-from-nas-synology/153730)
**Author:** [@Rahmat\_Agung\_W](https://discuss.elastic.co/u/Rahmat_Agung_W)\ **Replies:** 1\ **Last updated:** [October 24, 2018, 9:50am UTC](https://discuss.elastic.co/t/grok-filter-from-nas-synology/153730 "2018-10-24T09:50:38Z")
I have no clue how to filter this line log. Maybe someone wanna help. User \[guest\] from \[S07(192.168.99.2)\] via \[CIFS(SMB2)\] accessed shared folder \[GAME\]. User \[guest\] from \[L33(192.168.99.7)\] via \[CIFS(SMB2)\] accessed… --- ## [Logstash 6.4.1 Docker mutate split issue](https://discuss.elastic.co/t/logstash-6-4-1-docker-mutate-split-issue/153760)
**Author:** [@elastic3](https://discuss.elastic.co/u/elastic3)\ **Replies:** 0\ **Last updated:** [October 24, 2018, 9:28am UTC](https://discuss.elastic.co/t/logstash-6-4-1-docker-mutate-split-issue/153760 "2018-10-24T09:28:23Z")
Hi, I am trying to use mutate split on '\\t' (tab) but it only splits once as shown below. I even tried to convert it to string using mutate -convert field to string. Is it the double quotes? I even tried to escape the b… --- ## [Logstash error attempted to send a bulk request](https://discuss.elastic.co/t/logstash-error-attempted-to-send-a-bulk-request/153722)
**Author:** [@xUmaRix](https://discuss.elastic.co/u/xUmaRix)\ **Replies:** 0\ **Last updated:** [October 24, 2018, 6:02am UTC](https://discuss.elastic.co/t/logstash-error-attempted-to-send-a-bulk-request/153722 "2018-10-24T06:02:43Z")
OS DISTRIB\_ID=Ubuntu DISTRIB\_RELEASE=18.04 DISTRIB\_CODENAME=bionic DISTRIB\_DESCRIPTION="Ubuntu 18.04.1 LTS" NAME="Ubuntu" VERSION="18.04.1 LTS (Bionic Beaver)" ID=ubuntu ID\_LIKE=debian PRETTY\_NAME="Ubuntu 18.04.1 LTS" V… --- ## [Multiple Pipeline now working](https://discuss.elastic.co/t/multiple-pipeline-now-working/153720)
**Author:** [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)\ **Replies:** 1\ **Last updated:** [October 24, 2018, 5:59am UTC](https://discuss.elastic.co/t/multiple-pipeline-now-working/153720 "2018-10-24T05:59:24Z")
Hi, I am trying to configure logstash with multiple pipelines. Beats and Syslog but only Beats pipeline is working. \[2018-10-24T10:55:13,949\]\[INFO \]\[logstash.runner \] Starting Logstash {"logstash.version"=\>"6.… --- ## [Only connect to rabbitmq user username 'guest' and vhost '/'](https://discuss.elastic.co/t/only-connect-to-rabbitmq-user-username-guest-and-vhost/153563)
**Author:** [@Jenyow](https://discuss.elastic.co/u/Jenyow)\ **Replies:** 2\ **Last updated:** [October 24, 2018, 3:21am UTC](https://discuss.elastic.co/t/only-connect-to-rabbitmq-user-username-guest-and-vhost/153563 "2018-10-24T03:21:59Z")
Logstash version: 6.4.2 RabbitMQ version: 3.7.7 My rabbitmq input configuration and rabbitmq users is below: input { rabbitmq { type =\>"log" host =\> "127.0.0.1" port =\> 5682 vhost =\> "/" user =\> "guest" passw… --- ## [Logstash input file to es](https://discuss.elastic.co/t/logstash-input-file-to-es/152466)
**Author:** [@jianming](https://discuss.elastic.co/u/jianming)\ **Replies:** 5\ **Last updated:** [October 24, 2018, 3:24am UTC](https://discuss.elastic.co/t/logstash-input-file-to-es/152466 "2018-10-24T03:24:27Z")
i test a file output to ES, and i use "/usr/share/logstash/bin/logstash -f testlog.conf", the data output to ES, it's working and i Ctrl+c , and systemctl restart logstash. the data no output to ES. i don't know what … --- ## [Keep double quote on JSON trace](https://discuss.elastic.co/t/keep-double-quote-on-json-trace/153691)
**Author:** [@Jonathan\_Wadin](https://discuss.elastic.co/u/Jonathan_Wadin)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 10:11pm UTC](https://discuss.elastic.co/t/keep-double-quote-on-json-trace/153691 "2018-10-23T22:11:06Z")
Hello, I have a log file which produces a json content on one line well formatted. When I print it with the output { stdout { codec =\> rubydebug } } I see the correct Json with the fields double quote surround correctly… --- ## [Logstash unable to start pipeline:main (jvm.options issue?)](https://discuss.elastic.co/t/logstash-unable-to-start-pipeline-main-jvm-options-issue/153601)
**Author:** [@phoenix66](https://discuss.elastic.co/u/phoenix66)\ **Replies:** 2\ **Last updated:** [October 23, 2018, 9:09pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-pipeline-main-jvm-options-issue/153601 "2018-10-23T21:09:43Z")
Hi all, First post, so apologies if I miss any etiquette. I have recently installed Elastic Stack (6.4) and am having problems after configuring for Elastiflow. I get the following error in LS log: \[2018-10-23T13:31:1… --- ## [Please suggest me the filter/expression to be used for the event to metric data format conversion](https://discuss.elastic.co/t/please-suggest-me-the-filter-expression-to-be-used-for-the-event-to-metric-data-format-conversion/153683)
**Author:** [@ArunGan](https://discuss.elastic.co/u/ArunGan)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 9:05pm UTC](https://discuss.elastic.co/t/please-suggest-me-the-filter-expression-to-be-used-for-the-event-to-metric-data-format-conversion/153683 "2018-10-23T21:05:55Z")
I am trying to convert my event logs into influx protocol format using GROK filter. So i am trying to come up with a GROK filter to convert the below log . For example:, Input event log(From a node to log-stash) : 55.… --- ## [Expected Behavior When \`flush\_interval\` = \`clear\_interval\`?](https://discuss.elastic.co/t/expected-behavior-when-flush-interval-clear-interval/153681)
**Author:** [@Alsheh](https://discuss.elastic.co/u/Alsheh)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 8:55pm UTC](https://discuss.elastic.co/t/expected-behavior-when-flush-interval-clear-interval/153681 "2018-10-23T20:55:29Z")
In the metrics filter, what is the expected behavior when flush\_interval is equal to clear\_interval? Is there any possible race condition behavior here where counters get reset before events get flushed? filter { metr… --- ## [Logstash grok filter not creating fields in ES](https://discuss.elastic.co/t/logstash-grok-filter-not-creating-fields-in-es/153556)
**Author:** [@nsivabalan](https://discuss.elastic.co/u/nsivabalan)\ **Replies:** 1\ **Last updated:** [October 23, 2018, 7:16pm UTC](https://discuss.elastic.co/t/logstash-grok-filter-not-creating-fields-in-es/153556 "2018-10-23T19:16:36Z")
Hi, Can someone help with grok pattern for following log format? I had written my own with help of grok debugger but somehow it's not creating the fields in the ES. Error log: \[Sun Sep 09 04:00:36.495025 2018\] \[ssl:inf… --- ## [Swap the key and value in a kv pair](https://discuss.elastic.co/t/swap-the-key-and-value-in-a-kv-pair/153477)
**Author:** [@jcrossno](https://discuss.elastic.co/u/jcrossno)\ **Replies:** 0\ **Last updated:** [October 22, 2018, 8:29pm UTC](https://discuss.elastic.co/t/swap-the-key-and-value-in-a-kv-pair/153477 "2018-10-22T20:29:59Z")
I am receiving XML data which contains an array. The array can be cleaned up with gsub so it looks like a key=value pair, then parsed accordingly. My issue is that the pairs are in data:name order, so backwards for using… --- ## [\[Bug\] Metric Filter Events Ignore Conditionals](https://discuss.elastic.co/t/bug-metric-filter-events-ignore-conditionals/153656)
**Author:** [@Alsheh](https://discuss.elastic.co/u/Alsheh)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 5:13pm UTC](https://discuss.elastic.co/t/bug-metric-filter-events-ignore-conditionals/153656 "2018-10-23T17:13:22Z")
Events generated by the metrics filter ignore conditionals and execute all filters after the metrics filter. Logstash version: 6.4.2 logstash.conf: input { tcp { port =\> 11224 } } filter { kv { } if 'true'… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=660) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=662)