# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=680 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 681 --- ## [Get Data from XML WebService](https://discuss.elastic.co/t/get-data-from-xml-webservice/149830)
**Author:** [@Android\_Games](https://discuss.elastic.co/u/Android_Games)\ **Replies:** 2\ **Last updated:** [September 26, 2018, 5:29am UTC](https://discuss.elastic.co/t/get-data-from-xml-webservice/149830 "2018-09-26T05:29:03Z")
I'm new to Logstash and trying to build a dashboard,which will consume data webservice . I've installed elastic,Kibana and trying to configure logstash using http poller for pulling data from webservice I'm trying to a… --- ## [Logstash not flushing file to output directory](https://discuss.elastic.co/t/logstash-not-flushing-file-to-output-directory/149555)
**Author:** [@shwetabhsuman](https://discuss.elastic.co/u/shwetabhsuman)\ **Replies:** 1\ **Last updated:** [September 26, 2018, 4:38am UTC](https://discuss.elastic.co/t/logstash-not-flushing-file-to-output-directory/149555 "2018-09-26T04:38:41Z")
Hi, I am trying logstash in windows filesystem where I am trying to put file from one directory to other directory. My logstash.conf file is : - input { file { path =\> "C:/logstash/in.log" } } output { file { p… --- ## ["Detected ambiguous Field Reference" warnings in Logstash 6.4.0](https://discuss.elastic.co/t/detected-ambiguous-field-reference-warnings-in-logstash-6-4-0/149825)
**Author:** [@vicpav](https://discuss.elastic.co/u/vicpav)\ **Replies:** 1\ **Last updated:** [September 26, 2018, 1:05am UTC](https://discuss.elastic.co/t/detected-ambiguous-field-reference-warnings-in-logstash-6-4-0/149825 "2018-09-26T01:05:05Z")
Hi, After upgrading ELK cluster to 6.4.0 version, we have started to get Logstash warnings that make me puzzled: {"level":"WARN","loggerName":"org.logstash.FieldReference","timeMillis":1537825989853,"thread":"Ruby-0-Th… --- ## [Data replication from one ES to another using Logstash](https://discuss.elastic.co/t/data-replication-from-one-es-to-another-using-logstash/149690)
**Author:** [@Dolph\_2709](https://discuss.elastic.co/u/Dolph_2709)\ **Replies:** 3\ **Last updated:** [September 25, 2018, 9:54pm UTC](https://discuss.elastic.co/t/data-replication-from-one-es-to-another-using-logstash/149690 "2018-09-25T21:54:22Z")
Hello, I need to replicate data from old ES (ver. 1.1) instance to latest ELK. The older ES is part of IBM Mobile First solution, I can access it through REST API. This is a Logstash's config script I use on destinati… --- ## [Loading csv using logstash into Elastic](https://discuss.elastic.co/t/loading-csv-using-logstash-into-elastic/149605)
**Author:** [@Barath95](https://discuss.elastic.co/u/Barath95)\ **Replies:** 4\ **Last updated:** [September 25, 2018, 3:14pm UTC](https://discuss.elastic.co/t/loading-csv-using-logstash-into-elastic/149605 "2018-09-25T15:14:28Z")
Hi, I am using Logstash version 6.4.1 and trying to load a csv file into Elastic via Logstash. I am unable to load the file and keep getting the following error. \[2018-09-23T09:01:16,937\]\[ERROR\]\[logstash.pipeline … --- ## [Documents not getting indexed with custom timestamp](https://discuss.elastic.co/t/documents-not-getting-indexed-with-custom-timestamp/149828)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 0\ **Last updated:** [September 25, 2018, 12:55pm UTC](https://discuss.elastic.co/t/documents-not-getting-indexed-with-custom-timestamp/149828 "2018-09-25T12:55:17Z")
HI All, We are trying to index data using a custom timestamp and ended up with failure. Requesting for suggestions. If i index the data through @timestamp or without time filter i'm getting all the documents indexed, b… --- ## [Logstash use 100% CPU in lazy mode](https://discuss.elastic.co/t/logstash-use-100-cpu-in-lazy-mode/148314)
**Author:** [@max583](https://discuss.elastic.co/u/max583)\ **Replies:** 2\ **Last updated:** [September 25, 2018, 2:53pm UTC](https://discuss.elastic.co/t/logstash-use-100-cpu-in-lazy-mode/148314 "2018-09-25T14:53:16Z")
Good day! I have Logstash 6.2 with ES 6.2. Logstash use 100% of one core in lazy mode without any load. Is it normal? \[elog@srv7-owltank current\]$ ps aux | grep logstash elog 21660 110 3.4 5867628 1118488 pt… --- ## [Unable to send the csv file to elastic search and neither I am able to get the output on console](https://discuss.elastic.co/t/unable-to-send-the-csv-file-to-elastic-search-and-neither-i-am-able-to-get-the-output-on-console/149086)
**Author:** [@Prakhar\_Sandhu](https://discuss.elastic.co/u/Prakhar_Sandhu)\ **Replies:** 1\ **Last updated:** [September 25, 2018, 2:11pm UTC](https://discuss.elastic.co/t/unable-to-send-the-csv-file-to-elastic-search-and-neither-i-am-able-to-get-the-output-on-console/149086 "2018-09-25T14:11:13Z")
This is the logstash configuration file --- input { file { path =\> "C:\\Users\\pr389076\\Prakhar\\cars.csv" start\_position =\>"beginning" sincedb\_path =\> "NUL" } } filter { csv { separator =\> "," columns =\> \[ "make… --- ## [Logstash parses log files again: is that recurrent?](https://discuss.elastic.co/t/logstash-parses-log-files-again-is-that-recurrent/149812)
**Author:** [@Michel99\_7](https://discuss.elastic.co/u/Michel99_7)\ **Replies:** 4\ **Last updated:** [September 25, 2018, 2:06pm UTC](https://discuss.elastic.co/t/logstash-parses-log-files-again-is-that-recurrent/149812 "2018-09-25T14:06:51Z")
Hi all, Our environment is the following: OS: Red Hat Enterprise Linux Server release 7.3 Logstash version: 6.3.0 ES version: 6.3.2 Our problem: after having stopped the Logstash process for modification purpose in t… --- ## [JDBC input plugin](https://discuss.elastic.co/t/jdbc-input-plugin/149070)
**Author:** [@Goyal.vikas87](https://discuss.elastic.co/u/Goyal.vikas87)\ **Replies:** 1\ **Last updated:** [September 25, 2018, 2:03pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin/149070 "2018-09-25T14:03:18Z")
I am using below configuration in jdbc input: Use\_column\_value = true Tracking\_column = end\_dt (timestamp field) I have some records in table where end\_dt value is null showing that job is in running status. But when … --- ## [JSON parse error, original data now in message field, even though its a valid json](https://discuss.elastic.co/t/json-parse-error-original-data-now-in-message-field-even-though-its-a-valid-json/149033)
**Author:** [@gopalkakularam](https://discuss.elastic.co/u/gopalkakularam)\ **Replies:** 1\ **Last updated:** [September 25, 2018, 1:26pm UTC](https://discuss.elastic.co/t/json-parse-error-original-data-now-in-message-field-even-though-its-a-valid-json/149033 "2018-09-25T13:26:57Z")
This is the json log posting to logstash: { "@version" : 1, "source\_host" : "Gopals-MacBook-Pro-3.local", "message" : "Loading source class org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfigurat… --- ## [Logstash service starts when i run /bin/logstash-plugin install command](https://discuss.elastic.co/t/logstash-service-starts-when-i-run-bin-logstash-plugin-install-command/149836)
**Author:** [@Miodrag\_Lazarevic](https://discuss.elastic.co/u/Miodrag_Lazarevic)\ **Replies:** 0\ **Last updated:** [September 25, 2018, 1:25pm UTC](https://discuss.elastic.co/t/logstash-service-starts-when-i-run-bin-logstash-plugin-install-command/149836 "2018-09-25T13:25:43Z")
I' ve installed logstash service and I want to install necessary plugins, but when i run /usr/share/logstash/bin/logstash-plugin install logstash-output-amazon\_es Instead of installing amazon\_es plugin, it starts logst… --- ## [Logstash using custom Java](https://discuss.elastic.co/t/logstash-using-custom-java/149763)
**Author:** [@serozhka](https://discuss.elastic.co/u/serozhka)\ **Replies:** 3\ **Last updated:** [September 25, 2018, 11:41am UTC](https://discuss.elastic.co/t/logstash-using-custom-java/149763 "2018-09-25T11:41:42Z")
Hi guys! I have a Linux (RHEL) server, where I have to use logstash 6 in order to send log files to Kafka. Unfortunately, I dont have sudo there, also it has Java 1.7 and it is difficult to upgrade since these are produ… --- ## [Logstash 6.4.0 reload config - queue failed to obtain exclusive access](https://discuss.elastic.co/t/logstash-6-4-0-reload-config-queue-failed-to-obtain-exclusive-access/146493)
**Author:** [@paltryeffort](https://discuss.elastic.co/u/paltryeffort)\ **Replies:** 3\ **Last updated:** [September 25, 2018, 10:53am UTC](https://discuss.elastic.co/t/logstash-6-4-0-reload-config-queue-failed-to-obtain-exclusive-access/146493 "2018-09-25T10:53:00Z")
I have Logstash 6.4.0 running with one persistent queue "main" and 8 pipeline workers. When sending SIGHUP I get an error. Restarting the service works fine. \[2018-08-29T08:38:51,873\]\[WARN \]\[logstash.runner \] S… --- ## [Issue on setting up Logstash](https://discuss.elastic.co/t/issue-on-setting-up-logstash/149679)
**Author:** [@pananth](https://discuss.elastic.co/u/pananth)\ **Replies:** 17\ **Last updated:** [September 25, 2018, 10:02am UTC](https://discuss.elastic.co/t/issue-on-setting-up-logstash/149679 "2018-09-25T10:02:52Z")
I am facing issue on configuring Logstash, have set java\_home with Jave 1.8.0\_171 (JDK) While running logstash -f logstash-sample.conf - throwing error as Oracle "" was unexpected at this time. On other side used N… --- ## [Index exchange server and eml files](https://discuss.elastic.co/t/index-exchange-server-and-eml-files/149793)
**Author:** [@elitzur\_e](https://discuss.elastic.co/u/elitzur_e)\ **Replies:** 0\ **Last updated:** [September 25, 2018, 9:50am UTC](https://discuss.elastic.co/t/index-exchange-server-and-eml-files/149793 "2018-09-25T09:50:48Z")
Hi guys. i been looking for the right solution to get the messages of a certain mail box on exchange server into elastic. Seems that logstash might be what i need (current me if im wrong). is there a plugin for this? or… --- ## [Migrating Mysql data to elasticsearch using logstash](https://discuss.elastic.co/t/migrating-mysql-data-to-elasticsearch-using-logstash/149395)
**Author:** [@padmaja](https://discuss.elastic.co/u/padmaja)\ **Replies:** 7\ **Last updated:** [September 25, 2018, 7:02am UTC](https://discuss.elastic.co/t/migrating-mysql-data-to-elasticsearch-using-logstash/149395 "2018-09-25T07:02:04Z")
Hi, I'm new to kibana. i need a brief explanation of how can i convert MySQL data to Elastic Search .I have just installed all the requirements and tried to convert data using jdbc input plugin but, i'm unable to do it.… --- ## [Grok can't match my log](https://discuss.elastic.co/t/grok-cant-match-my-log/149753)
**Author:** [@Haitianisgood](https://discuss.elastic.co/u/Haitianisgood)\ **Replies:** 0\ **Last updated:** [September 25, 2018, 3:20am UTC](https://discuss.elastic.co/t/grok-cant-match-my-log/149753 "2018-09-25T03:20:54Z")
My log: \[23:59:52.743\]2018-08-14 23:59:52 DEBUG \[resin-tcp-connection-\*:8012-59\] /queryup\[sversion=5\]\[model=HUAWEI G7-UL20\]\[from=\_360\]\[lastupdatetime=1532506757000\]\[versioncode=201805140\]\[userId=13994251\]\[mac=8638460274… --- ## [Change JVM Heap Size with Logstash Helm Chart](https://discuss.elastic.co/t/change-jvm-heap-size-with-logstash-helm-chart/149736)
**Author:** [@chenchik](https://discuss.elastic.co/u/chenchik)\ **Replies:** 2\ **Last updated:** [September 24, 2018, 10:02pm UTC](https://discuss.elastic.co/t/change-jvm-heap-size-with-logstash-helm-chart/149736 "2018-09-24T22:02:43Z")
I'm currently using the Logstash Kubernetes Helm chart, linked below: There doesn't seem to be a way to increase the JVM heap size using the values.yaml file. I have also tried setting the environment variable LS\_JAV… --- ## [New user questions about date format](https://discuss.elastic.co/t/new-user-questions-about-date-format/149708)
**Author:** [@Fpig](https://discuss.elastic.co/u/Fpig)\ **Replies:** 1\ **Last updated:** [September 24, 2018, 10:00pm UTC](https://discuss.elastic.co/t/new-user-questions-about-date-format/149708 "2018-09-24T22:00:23Z")
I have two inputs pulling in data in different formats. My problem is the timestamps are causing issues. The first looks like something this 2018-09-21T22:13:27.161Z and the second will have this format 4/11/18 17:19 H… --- ## [Logstash stops to receive new inputs after 5 minutes](https://discuss.elastic.co/t/logstash-stops-to-receive-new-inputs-after-5-minutes/148685)
**Author:** [@Woolis\_Xiibil\_Peek](https://discuss.elastic.co/u/Woolis_Xiibil_Peek)\ **Replies:** 2\ **Last updated:** [September 24, 2018, 9:15pm UTC](https://discuss.elastic.co/t/logstash-stops-to-receive-new-inputs-after-5-minutes/148685 "2018-09-24T21:15:05Z")
Hi. I having trouble with some inputs from filebeats. Original I was receiving inputs from tree differents sources in three different ports (5044, 5045, and 5046), each source in one port. Today, we had add 6 more ser… --- ## [Improve performance in a single-node deployment](https://discuss.elastic.co/t/improve-performance-in-a-single-node-deployment/149735)
**Author:** [@Rakios](https://discuss.elastic.co/u/Rakios)\ **Replies:** 0\ **Last updated:** [September 24, 2018, 8:37pm UTC](https://discuss.elastic.co/t/improve-performance-in-a-single-node-deployment/149735 "2018-09-24T20:37:04Z")
Hi, I hope you can help me. I'm currently trying to load a lot of logs that are almost xml. The problem comes because in a minute 20 logs files of 5mb are generated and logstash takes approximately 12 minutes to load th… --- ## [How to get in memory queue statiscic](https://discuss.elastic.co/t/how-to-get-in-memory-queue-statiscic/148995)
**Author:** [@dorinand](https://discuss.elastic.co/u/dorinand)\ **Replies:** 1\ **Last updated:** [September 24, 2018, 8:20pm UTC](https://discuss.elastic.co/t/how-to-get-in-memory-queue-statiscic/148995 "2018-09-24T20:20:52Z")
I used persistent queue for my logstash servers. I could check actual stat of queue via: curl -XGET 'localhost:9600/\_node/stats/ and get response: "queue": { "events": 0, "type": "persisted", "capacity": … --- ## [Fetching data from DB using logstash](https://discuss.elastic.co/t/fetching-data-from-db-using-logstash/149532)
**Author:** [@baddy9459](https://discuss.elastic.co/u/baddy9459)\ **Replies:** 1\ **Last updated:** [September 24, 2018, 6:27pm UTC](https://discuss.elastic.co/t/fetching-data-from-db-using-logstash/149532 "2018-09-24T18:27:43Z")
I am trying to fetch data from Database. I have field timedetail in database table, this field is in format "dd-MMM-yy HH.mm.ss.SSSS a" i want to add this field as @timestamp when i used filter{ date{ match=\> \["ti… --- ## [Xml filter - how to view/ output json as logstash sees it](https://discuss.elastic.co/t/xml-filter-how-to-view-output-json-as-logstash-sees-it/149485)
**Author:** [@safehouse](https://discuss.elastic.co/u/safehouse)\ **Replies:** 1\ **Last updated:** [September 24, 2018, 6:25pm UTC](https://discuss.elastic.co/t/xml-filter-how-to-view-output-json-as-logstash-sees-it/149485 "2018-09-24T18:25:53Z")
I'm using the xml filter and a ruby script to access nested nodes. What I am no clear on is the exact structure of the json which the ruby script will be acecssing, so I'm not sure how to write the event.get statement. F… --- ## [Getting parsing errors for csv filter](https://discuss.elastic.co/t/getting-parsing-errors-for-csv-filter/149440)
**Author:** [@mohsin\_khan\_adp](https://discuss.elastic.co/u/mohsin_khan_adp)\ **Replies:** 1\ **Last updated:** [September 24, 2018, 6:25pm UTC](https://discuss.elastic.co/t/getting-parsing-errors-for-csv-filter/149440 "2018-09-24T18:25:12Z")
Below is my header and record i am trying to push to elastic search. I am getting getting parsing errors and i could not ship it. can you help me here ? HEADER1|HEADER2|HEADER3|HEADER4|HEADER5|HEADER6|HEADER7|HEADER8|H… --- ## [Failed when parsing JSON from VirusTotal](https://discuss.elastic.co/t/failed-when-parsing-json-from-virustotal/149387)
**Author:** [@Rahmat\_Agung\_W](https://discuss.elastic.co/u/Rahmat_Agung_W)\ **Replies:** 1\ **Last updated:** [September 24, 2018, 6:23pm UTC](https://discuss.elastic.co/t/failed-when-parsing-json-from-virustotal/149387 "2018-09-24T18:23:28Z")
So I have JSON file retrieve from virustotal. Looks: This is my logstash config : input{ file{ path =\> "/path/scan.json" start\_position =\> beginning } } filter{ json{ source =\> scans } } output { elas… --- ## [Regular expression on array/list field](https://discuss.elastic.co/t/regular-expression-on-array-list-field/149329)
**Author:** [@jprat](https://discuss.elastic.co/u/jprat)\ **Replies:** 1\ **Last updated:** [September 24, 2018, 6:21pm UTC](https://discuss.elastic.co/t/regular-expression-on-array-list-field/149329 "2018-09-24T18:21:17Z")
Hi, I was wondering if it is possible to make an expression into an if condition where I could check if a field, which is an array/list, could contain any string matching certain regular expression. For example tags =\> … --- ## [Kv plugin behavior](https://discuss.elastic.co/t/kv-plugin-behavior/149304)
**Author:** [@alan.frabutt](https://discuss.elastic.co/u/alan.frabutt)\ **Replies:** 4\ **Last updated:** [September 24, 2018, 3:53pm UTC](https://discuss.elastic.co/t/kv-plugin-behavior/149304 "2018-09-24T15:53:56Z")
Greetings all, I believe I've made a solid effort digging around other posts (and sites) for more info, but I'm not getting traction. The behavior I'd expect from the code below is that it would break out each kv pair … --- ## [How to extract differnt log lines which are occuring between 2 lines of log files (Multiple grok with if condition)](https://discuss.elastic.co/t/how-to-extract-differnt-log-lines-which-are-occuring-between-2-lines-of-log-files-multiple-grok-with-if-condition/148934)
**Author:** [@richagautam](https://discuss.elastic.co/u/richagautam)\ **Replies:** 1\ **Last updated:** [September 24, 2018, 3:43pm UTC](https://discuss.elastic.co/t/how-to-extract-differnt-log-lines-which-are-occuring-between-2-lines-of-log-files-multiple-grok-with-if-condition/148934 "2018-09-24T15:43:28Z")
Hi Team, I am trying to parse multiple log files where log syntax are not same and extract different lines which is coming between 2 lines of log data. Below is the sample data Feb 12 18:24:26 localhost.localdomain: SA… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=679) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=681)