# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=738 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 739 --- ## [XPath issue with different XML files](https://discuss.elastic.co/t/xpath-issue-with-different-xml-files/135396)
**Author:** [@AurelienG](https://discuss.elastic.co/u/AurelienG)\ **Replies:** 7\ **Last updated:** [June 19, 2018, 2:39pm UTC](https://discuss.elastic.co/t/xpath-issue-with-different-xml-files/135396 "2018-06-19T14:39:32Z")
Hello, Here's my problem: I have several XML files, which don't have the same structure, and I need to get one of the fields, which is not even always there... So here's what I've tried: #Check if there's a jdk field i… --- ## [Combine log using logstash](https://discuss.elastic.co/t/combine-log-using-logstash/136391)
**Author:** [@dqtho](https://discuss.elastic.co/u/dqtho)\ **Replies:** 2\ **Last updated:** [June 19, 2018, 2:18pm UTC](https://discuss.elastic.co/t/combine-log-using-logstash/136391 "2018-06-19T14:18:53Z")
Hi all, I encounter with problem about zimbra mail log. When a user send email, zimbra create 2 log. First log with some field: email\_id, sender, receiver, subject.... Second log with field: email\_id, status (ok if… --- ## [Format json](https://discuss.elastic.co/t/format-json/135529)
**Author:** [@joce455](https://discuss.elastic.co/u/joce455)\ **Replies:** 4\ **Last updated:** [June 19, 2018, 2:17pm UTC](https://discuss.elastic.co/t/format-json/135529 "2018-06-19T14:17:51Z")
Hi all, i have a log in json format: \[{"id":"917066d5-5244-42e1-9343-49ccefe5b8cb","name":"data1","dataamount":1024000,"notification":0}\] And I want to transform in this: id:917066d5-5244-42e1-9343-49ccefe5b8cb name:… --- ## [Why am I getting \_dateparsefailure in Logstash?](https://discuss.elastic.co/t/why-am-i-getting-dateparsefailure-in-logstash/136484)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 10\ **Last updated:** [June 19, 2018, 1:42pm UTC](https://discuss.elastic.co/t/why-am-i-getting-dateparsefailure-in-logstash/136484 "2018-06-19T13:42:45Z")
Hi, I am trying out ELK GA 6.3. Using Logstash jdbc plugin, I am fetching rows from my oracle table. Below is my complete configuration; input { jdbc { jdbc\_driver\_library =\> "/Logstash/ojdbc6.jar" jdbc\_driver\_cla… --- ## [Split a string by commas but ignore commas within double-quotes？](https://discuss.elastic.co/t/split-a-string-by-commas-but-ignore-commas-within-double-quotes/136471)
**Author:** [@Huang\_Gyo](https://discuss.elastic.co/u/Huang_Gyo)\ **Replies:** 1\ **Last updated:** [June 19, 2018, 1:13pm UTC](https://discuss.elastic.co/t/split-a-string-by-commas-but-ignore-commas-within-double-quotes/136471 "2018-06-19T13:13:17Z")
For example, my log is like: 2018-06-12T10:58:19.617Z, , "test,test", BJSGEXMBX12, , , SMTP, HARECEIVE, 41661182775666 How to split it by "," but ignore the "," in "test,test"? Thanks! --- ## [Logstash constantly restarting](https://discuss.elastic.co/t/logstash-constantly-restarting/135972)
**Author:** [@jintawk](https://discuss.elastic.co/u/jintawk)\ **Replies:** 2\ **Last updated:** [June 19, 2018, 11:21am UTC](https://discuss.elastic.co/t/logstash-constantly-restarting/135972 "2018-06-19T11:21:26Z")
Hi, very new to ELK stack & just trying trying to set everything up on a fresh install of Ubuntu server 16.04 with openjdk8 I have a C# application with NLog which is posting logs to the elasticsearch endpoint on the Ub… --- ## [Additional console logging in 6.3.0](https://discuss.elastic.co/t/additional-console-logging-in-6-3-0/135979)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 3\ **Last updated:** [June 19, 2018, 11:02am UTC](https://discuss.elastic.co/t/additional-console-logging-in-6-3-0/135979 "2018-06-19T11:02:00Z")
The release notes do not mention it, but having upgraded to 6.3.0 logstash is now producing additional logging to stdout. This appears to me to be the effect of 9076. Looking at the new log4j2.properties there are a coup… --- ## [Elasticsearch listen on ipv4 but logstash won't?](https://discuss.elastic.co/t/elasticsearch-listen-on-ipv4-but-logstash-wont/136475)
**Author:** [@markdaku](https://discuss.elastic.co/u/markdaku)\ **Replies:** 0\ **Last updated:** [June 19, 2018, 10:57am UTC](https://discuss.elastic.co/t/elasticsearch-listen-on-ipv4-but-logstash-wont/136475 "2018-06-19T10:57:59Z")
This topic has been tossed around several times. Both Elasticsearch and Logstash are Java apps. Why is that elastic search can listen on ipv4 out of the box and logstash listens to ipv6 on the same host? Every single … --- ## [Logstash grok on dynamic JSON object](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408)
**Author:** [@pohsun.teh](https://discuss.elastic.co/u/pohsun.teh)\ **Replies:** 8\ **Last updated:** [June 19, 2018, 9:53am UTC](https://discuss.elastic.co/t/logstash-grok-on-dynamic-json-object/136408 "2018-06-19T09:53:32Z")
Greetings, I have a dynamic JSON object which I would like to extract fields from it This is sample on how the JSON object looks like layers: { tcap: \[ { otid: "12:23:34:45" }, { otid: "4h:2f… --- ## [Best practice to handle logstash configuratioon](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977)
**Author:** [@tag\_v](https://discuss.elastic.co/u/tag_v)\ **Replies:** 9\ **Last updated:** [June 19, 2018, 9:51am UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977 "2018-06-19T09:51:17Z")
Hello All, My logstash configuration have almost 1500+ lines of code where i am taking input from beats and applying filters for extracting fields of various log types and output to kafka. Is there any way to handle hug… --- ## [Logstash elasticsearch filter plugin exact match](https://discuss.elastic.co/t/logstash-elasticsearch-filter-plugin-exact-match/136308)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 1\ **Last updated:** [June 19, 2018, 9:01am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-filter-plugin-exact-match/136308 "2018-06-19T09:01:47Z")
Hi, I'm using elasticsearch filter plugin in a pipeline configuration like this: elasticsearch { hosts =\> \["elasticsearch:9200"\] index =\> \["logstash-indexname"\] query =\> "item:%{\[itemname\]}" The question is: how can… --- ## [Logstash index diretory vs files](https://discuss.elastic.co/t/logstash-index-diretory-vs-files/136449)
**Author:** [@aminhk](https://discuss.elastic.co/u/aminhk)\ **Replies:** 1\ **Last updated:** [June 19, 2018, 8:43am UTC](https://discuss.elastic.co/t/logstash-index-diretory-vs-files/136449 "2018-06-19T08:43:40Z")
Hi guys, I have a question. I have 5 log files and 5 configs file for logstash. if i put all the .conf file in directory and start logstash to index that directory is better than calling each conf file separately(logst… --- ## [How to filter only below format from log file](https://discuss.elastic.co/t/how-to-filter-only-below-format-from-log-file/136414)
**Author:** [@Naveen\_Reddy\_Sama](https://discuss.elastic.co/u/Naveen_Reddy_Sama)\ **Replies:** 3\ **Last updated:** [June 19, 2018, 8:38am UTC](https://discuss.elastic.co/t/how-to-filter-only-below-format-from-log-file/136414 "2018-06-19T08:38:30Z")
Hi Folks, below log is getting from the JMeter log file, can anyone help how to write logstash config filter. timeStamp,elapsed,label,responseCode,threadName,success,bytes,sentBytes,grpThreads,allThreads,Latency,Sample… --- ## [How to use logstash to upload data in elasticsearch in specific format](https://discuss.elastic.co/t/how-to-use-logstash-to-upload-data-in-elasticsearch-in-specific-format/135642)
**Author:** [@rs123](https://discuss.elastic.co/u/rs123)\ **Replies:** 8\ **Last updated:** [June 19, 2018, 8:12am UTC](https://discuss.elastic.co/t/how-to-use-logstash-to-upload-data-in-elasticsearch-in-specific-format/135642 "2018-06-19T08:12:52Z")
Both mapping and index are as below: PUT somki { "mappings": { "\_doc" : { "properties" : { "suggest" : { "type" : "completion", "contexts"… --- ## [Logstash 6.3 requires x-pack security?](https://discuss.elastic.co/t/logstash-6-3-requires-x-pack-security/136326)
**Author:** [@Hermann.Hans](https://discuss.elastic.co/u/Hermann.Hans)\ **Replies:** 5\ **Last updated:** [June 19, 2018, 6:13am UTC](https://discuss.elastic.co/t/logstash-6-3-requires-x-pack-security/136326 "2018-06-19T06:13:14Z")
Hey everyone, So I just upgraded our ES cluster to version 6.3. All nodes are up and running, as well as Kibana. Unfortunately our logstash instances refuse to connect to elasticsearch now, claiming that X-Pack Security… --- ## [How to tell that logstash is idle](https://discuss.elastic.co/t/how-to-tell-that-logstash-is-idle/136409)
**Author:** [@shiranbi07](https://discuss.elastic.co/u/shiranbi07)\ **Replies:** 1\ **Last updated:** [June 19, 2018, 5:55am UTC](https://discuss.elastic.co/t/how-to-tell-that-logstash-is-idle/136409 "2018-06-19T05:55:28Z")
my input is a folder with log files I want to know that logstash completed going over all of the current data and is currently waiting for the files to be updated or more files to be created. In other words, it is curr… --- ## [How to write Customize grok with multiple json values?](https://discuss.elastic.co/t/how-to-write-customize-grok-with-multiple-json-values/135238)
**Author:** [@Harsh\_Sharma](https://discuss.elastic.co/u/Harsh_Sharma)\ **Replies:** 10\ **Last updated:** [June 19, 2018, 5:49am UTC](https://discuss.elastic.co/t/how-to-write-customize-grok-with-multiple-json-values/135238 "2018-06-19T05:49:11Z")
Below are the sample logs. I was able to write a customize grok on it and took 'onDPriceDetail' value in GREEDYDATA. But I want to parse this multiple child data of JSON format with different fields name. 2018-06-08 16:… --- ## [Nested JSON parsing in dynamic field names](https://discuss.elastic.co/t/nested-json-parsing-in-dynamic-field-names/136211)
**Author:** [@Somya\_Makkad](https://discuss.elastic.co/u/Somya_Makkad)\ **Replies:** 1\ **Last updated:** [June 18, 2018, 11:41pm UTC](https://discuss.elastic.co/t/nested-json-parsing-in-dynamic-field-names/136211 "2018-06-18T23:41:59Z")
{ "0": { "username": "rita", "age": 21, "occupation": "teacher" }, "1": { "username": "nita", "age": 21, "occupation": "police officer" } } My log format looks as above. When applying the json filter it parse… --- ## [Logstash timestamp with CEST](https://discuss.elastic.co/t/logstash-timestamp-with-cest/136312)
**Author:** [@ginu](https://discuss.elastic.co/u/ginu)\ **Replies:** 6\ **Last updated:** [June 18, 2018, 10:49pm UTC](https://discuss.elastic.co/t/logstash-timestamp-with-cest/136312 "2018-06-18T22:49:55Z")
Hi all I have a timestamp in the log which has the follwing format ",xxxxxxx,Thu May 31 09:40:46 CEST 2018,yyyyyy...." I tried to have the config ... ... filter { if (\[path\] =~ /log/) { csv { columns =\> \[ "even… --- ## [Method to handle with false positive files](https://discuss.elastic.co/t/method-to-handle-with-false-positive-files/136248)
**Author:** [@dyl](https://discuss.elastic.co/u/dyl)\ **Replies:** 2\ **Last updated:** [June 18, 2018, 10:39pm UTC](https://discuss.elastic.co/t/method-to-handle-with-false-positive-files/136248 "2018-06-18T22:39:48Z")
Hello ELK community. I would like to know the way to handle with false positive files. Let me explain. I use filebeat to listen a directory which contains several files. On these files, some of them are "false positive… --- ## [Logstash 6.3.0 Upgrade](https://discuss.elastic.co/t/logstash-6-3-0-upgrade/136243)
**Author:** [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\ **Replies:** 4\ **Last updated:** [June 18, 2018, 10:38pm UTC](https://discuss.elastic.co/t/logstash-6-3-0-upgrade/136243 "2018-06-18T22:38:20Z")
After upgrading elasticsearch logstash and kibana to 6.3.0 I have two new tabs in Kibana Metric and APM I've enabled the Metric and now I have stats for es and kibana, however logstash stats are not in there. From wha… --- ## [Upgrading to 6.3+ with Persistent Queues Enabled](https://discuss.elastic.co/t/upgrading-to-6-3-with-persistent-queues-enabled/136386)
**Author:** [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Replies:** 0\ **Last updated:** [June 18, 2018, 10:26pm UTC](https://discuss.elastic.co/t/upgrading-to-6-3-with-persistent-queues-enabled/136386 "2018-06-18T22:26:38Z")
Logstash 6.3.0 has landed, and with it comes some useful improvements to the Persistent Queue, an opt-in feature that enables disk-based spooling and resiliency across abnormal shutdowns. If you're using the Persistent … --- ## [Help Parsing JSONFMT Rsyslog entries](https://discuss.elastic.co/t/help-parsing-jsonfmt-rsyslog-entries/136359)
**Author:** [@james\_mtl](https://discuss.elastic.co/u/james_mtl)\ **Replies:** 2\ **Last updated:** [June 18, 2018, 9:05pm UTC](https://discuss.elastic.co/t/help-parsing-jsonfmt-rsyslog-entries/136359 "2018-06-18T21:05:30Z")
I am having difficulty parsing the below. Actual rsyslog line. 2018-06-18T12:36:55.588494-05:00 lb-workers.awsprod.com haproxy\[2639\]: 172.0.0.1:27947 \[18/Jun/2018:12:36:55.587\] application~ payment\_backend/\ 0/-1… --- ## [Logstash starts pipeline but no output on command line or index created](https://discuss.elastic.co/t/logstash-starts-pipeline-but-no-output-on-command-line-or-index-created/136364)
**Author:** [@AmaanMoledina](https://discuss.elastic.co/u/AmaanMoledina)\ **Replies:** 3\ **Last updated:** [June 18, 2018, 8:51pm UTC](https://discuss.elastic.co/t/logstash-starts-pipeline-but-no-output-on-command-line-or-index-created/136364 "2018-06-18T20:51:34Z")
Here is the Config file input { file { path =\> "c:/Users/Admin/Desktop/data/cars.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" ignore\_older =\> 0 } } fil… --- ## [Keep only a certain number of lines of the same type](https://discuss.elastic.co/t/keep-only-a-certain-number-of-lines-of-the-same-type/136244)
**Author:** [@dyl](https://discuss.elastic.co/u/dyl)\ **Replies:** 1\ **Last updated:** [June 18, 2018, 7:19pm UTC](https://discuss.elastic.co/t/keep-only-a-certain-number-of-lines-of-the-same-type/136244 "2018-06-18T19:19:19Z")
Hello ELK community. I use logstash and filebeat to parse files which are contains into a directory. For exemple, one of my log file looks like : INFO;0000;000001;\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*… --- ## [Logstash can't create pipelines](https://discuss.elastic.co/t/logstash-cant-create-pipelines/136336)
**Author:** [@Ruslan](https://discuss.elastic.co/u/Ruslan)\ **Replies:** 2\ **Last updated:** [June 18, 2018, 6:37pm UTC](https://discuss.elastic.co/t/logstash-cant-create-pipelines/136336 "2018-06-18T18:37:30Z")
I have a cluster of 3 elastic search nodes + kibana vm - version 6.2.4 running on 4 vms of Centos 7 . Logstash instances running on every elasticsearch nodes The logstash service is running, but any indices have been cr… --- ## [Pipe input plugin for a remote logfile](https://discuss.elastic.co/t/pipe-input-plugin-for-a-remote-logfile/136323)
**Author:** [@zeenon](https://discuss.elastic.co/u/zeenon)\ **Replies:** 1\ **Last updated:** [June 18, 2018, 6:35pm UTC](https://discuss.elastic.co/t/pipe-input-plugin-for-a-remote-logfile/136323 "2018-06-18T18:35:23Z")
Hi, I have to collect logs from a remote AIX server without permission to install a remote logstash So, is it correct to use the following command in the pipe input plugin ? command =\> "ssh myuser@rs6000 -c 'tail -f /… --- ## [Bulk messages sent through Logstash are stored a message](https://discuss.elastic.co/t/bulk-messages-sent-through-logstash-are-stored-a-message/134265)
**Author:** [@paulo.leao](https://discuss.elastic.co/u/paulo.leao)\ **Replies:** 4\ **Last updated:** [June 18, 2018, 5:54pm UTC](https://discuss.elastic.co/t/bulk-messages-sent-through-logstash-are-stored-a-message/134265 "2018-06-18T17:54:22Z")
Hello Everyone, I developed a powershell script to send message in bulk to Elastiseach using the Invoke-RestMethod command. When I send a bulk request directly to Elasticsearch, I can see the properties of my json obje… --- ## ["No current connection" from jdbc\_static](https://discuss.elastic.co/t/no-current-connection-from-jdbc-static/136108)
**Author:** [@cbarton](https://discuss.elastic.co/u/cbarton)\ **Replies:** 5\ **Last updated:** [June 18, 2018, 5:03pm UTC](https://discuss.elastic.co/t/no-current-connection-from-jdbc-static/136108 "2018-06-18T17:03:47Z")
Hello, I'm trying to set up a logstash pipeline that does an enrichment lookup via the jdbc\_static plugin, calling a MariaDB database via the MySQL jdbc connector. Aside - I tried doing it initially with the MariaDB con… --- ## [Logstash Input Pluging Misses Events](https://discuss.elastic.co/t/logstash-input-pluging-misses-events/136340)
**Author:** [@remziay](https://discuss.elastic.co/u/remziay)\ **Replies:** 0\ **Last updated:** [June 18, 2018, 4:16pm UTC](https://discuss.elastic.co/t/logstash-input-pluging-misses-events/136340 "2018-06-18T16:16:38Z")
Hello , I am using Logstash input plugin to process many files in order to import them to ES. Config is like below: input { file { path =\> "/path/\*.csv" start\_position =\> “beginning” } filter { csv { } } outpu… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=737) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=739)