# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=685 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 686 --- ## [Logstash not running](https://discuss.elastic.co/t/logstash-not-running/146925)
**Author:** [@rorobolla](https://discuss.elastic.co/u/rorobolla)\ **Replies:** 17\ **Last updated:** [September 15, 2018, 3:23pm UTC](https://discuss.elastic.co/t/logstash-not-running/146925 "2018-09-15T15:23:53Z")
I've been trying and trying to get logstash to run but it keeps coming up with the same error: ERROR: Pipelines YAML file is empty. Location: /Users/bollampally.r/Downloads/sre/logstash-6.4.0/config/pipelines.yml usage… --- ## [XML child nodes with varying number into one ElasticSearch field?](https://discuss.elastic.co/t/xml-child-nodes-with-varying-number-into-one-elasticsearch-field/148644)
**Author:** [@safehouse](https://discuss.elastic.co/u/safehouse)\ **Replies:** 6\ **Last updated:** [September 14, 2018, 6:46pm UTC](https://discuss.elastic.co/t/xml-child-nodes-with-varying-number-into-one-elasticsearch-field/148644 "2018-09-14T18:46:09Z")
I've searched the forum and tried various processes, but I can't get to a solution. Any help would be really appreciated. I have XML like this: \ \ **Author:** [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\ **Replies:** 1\ **Last updated:** [September 14, 2018, 6:21pm UTC](https://discuss.elastic.co/t/we-lost-some-metrics-from-time-to-time-blocked-by-logstash/148554 "2018-09-14T18:21:44Z") Dear Elastic ， We lost some metrics from time to time recently. Could someone give some suggestion to solve this problem? Error from Metricbeat 2018-09-14T11:03:37+08:00 ERR Failed to publish events caused by: EOF 20… --- ## [Create new columns from one column value](https://discuss.elastic.co/t/create-new-columns-from-one-column-value/148603)
**Author:** [@Varun\_Goyal](https://discuss.elastic.co/u/Varun_Goyal)\ **Replies:** 1\ **Last updated:** [September 14, 2018, 5:29pm UTC](https://discuss.elastic.co/t/create-new-columns-from-one-column-value/148603 "2018-09-14T17:29:45Z")
I have a csv columns as: Data1,Data2,"Product: XYZ\\r\\nVersion: 1.0.0\\r\\nLanguage: English\\r\\nDate: Wednesday, September 12, 2018\\r\\n" By using filter I am able to get data in columns, here is my filter: csv { #autode… --- ## [How to Force shutdown running Logstash in windows using command prompt](https://discuss.elastic.co/t/how-to-force-shutdown-running-logstash-in-windows-using-command-prompt/148648)
**Author:** [@gowt](https://discuss.elastic.co/u/gowt)\ **Replies:** 1\ **Last updated:** [September 14, 2018, 5:09pm UTC](https://discuss.elastic.co/t/how-to-force-shutdown-running-logstash-in-windows-using-command-prompt/148648 "2018-09-14T17:09:20Z")
Hi I am running logstash with '1' worker process to load data from sql to elastic using command line 'bin/logstash -f logstash.conf' on regular intervals. it is executing fin with no issues in positive scenarios. Somet… --- ## [Clair describes Security Vulnerability](https://discuss.elastic.co/t/clair-describes-security-vulnerability/148489)
**Author:** [@Jas\_Ahluwalia](https://discuss.elastic.co/u/Jas_Ahluwalia)\ **Replies:** 1\ **Last updated:** [September 14, 2018, 3:12pm UTC](https://discuss.elastic.co/t/clair-describes-security-vulnerability/148489 "2018-09-14T15:12:39Z")
Hi, I'm using 6.4.0 of the ELK stack in docker containers. We're using the clair vulnerability scanner to scan for vulnerabilities in our docker images. I see a high vulnerability for APM, Metricbeat, Logstash, Elastics… --- ## [Erros Logstash fields Kibana](https://discuss.elastic.co/t/erros-logstash-fields-kibana/146907)
**Author:** [@marcos.vbp](https://discuss.elastic.co/u/marcos.vbp)\ **Replies:** 6\ **Last updated:** [September 14, 2018, 3:10pm UTC](https://discuss.elastic.co/t/erros-logstash-fields-kibana/146907 "2018-09-14T15:10:16Z")
Hello, I'm try start with ELK, but very dificult for me. I try read log access.log for weblogic , output exemple is : 172.16.27.211 - - \[30/Ago/2018:10:05:41 -0300\] "GET /teste/rest/currentUser?noCache=1535634341625 … --- ## [Ingesting openLDAP logs](https://discuss.elastic.co/t/ingesting-openldap-logs/148638)
**Author:** [@DunkB](https://discuss.elastic.co/u/DunkB)\ **Replies:** 0\ **Last updated:** [September 14, 2018, 2:55pm UTC](https://discuss.elastic.co/t/ingesting-openldap-logs/148638 "2018-09-14T14:55:32Z")
Hi All, Is anyone ingesting openldap logs in a way that makes it easy to analyse what was done by a certain client, find all the clients failing TLS handshakes or binding as a certain user for example. client transacti… --- ## [In Logstash, how do I limit the depth of JSON properties in my logs that are turned into Index fields in Elasticsearch?](https://discuss.elastic.co/t/in-logstash-how-do-i-limit-the-depth-of-json-properties-in-my-logs-that-are-turned-into-index-fields-in-elasticsearch/147728)
**Author:** [@jlavallet](https://discuss.elastic.co/u/jlavallet)\ **Replies:** 1\ **Last updated:** [September 14, 2018, 2:36pm UTC](https://discuss.elastic.co/t/in-logstash-how-do-i-limit-the-depth-of-json-properties-in-my-logs-that-are-turned-into-index-fields-in-elasticsearch/147728 "2018-09-14T14:36:48Z")
I'm fairly new to the Elastic Stack. I'm using Logstash 6.4.0 to load JSON log data from Filebeat 6.4.0 into Elasticsearch 6.4.0.. I'm finding that I'm getting way too many JSON properties converted into fields once I st… --- ## [Mongo db to elastic 6.4](https://discuss.elastic.co/t/mongo-db-to-elastic-6-4/147912)
**Author:** [@Pirai\_Sudie](https://discuss.elastic.co/u/Pirai_Sudie)\ **Replies:** 2\ **Last updated:** [September 14, 2018, 10:00am UTC](https://discuss.elastic.co/t/mongo-db-to-elastic-6-4/147912 "2018-09-14T10:00:15Z")
How to sync data from MongoDB into elastic using logstash... There is no offical input plugin then how to solve this.... --- ## [Already used ports](https://discuss.elastic.co/t/already-used-ports/148560)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 1\ **Last updated:** [September 14, 2018, 9:53am UTC](https://discuss.elastic.co/t/already-used-ports/148560 "2018-09-14T09:53:43Z")
hello how to fix this err? Thanks in advance. \[2018-09-14T13:45:21,892\]\[ERROR\]\[logstash.pipeline \] A plugin had an unrecoverable error. Will restart this plugin. Pipeline\_id:main Plugin: \ **Author:** [@Arun\_Murugappan](https://discuss.elastic.co/u/Arun_Murugappan)\ **Replies:** 4\ **Last updated:** [September 14, 2018, 8:44am UTC](https://discuss.elastic.co/t/kv-parse/146064 "2018-09-14T08:44:58Z") Sample {"Time": "2018-05-10", "Server": {"Name": " Default , "id" : " 521"}} When i parse using kv with : and , i am getting time field right but not the name. I am gerting server: Name: Default but i just require Name… --- ## [Not able to ship logs from Collectd to logstash](https://discuss.elastic.co/t/not-able-to-ship-logs-from-collectd-to-logstash/148582)
**Author:** [@Abhimanue\_Rana](https://discuss.elastic.co/u/Abhimanue_Rana)\ **Replies:** 0\ **Last updated:** [September 14, 2018, 8:20am UTC](https://discuss.elastic.co/t/not-able-to-ship-logs-from-collectd-to-logstash/148582 "2018-09-14T08:20:32Z")
I have installed ELK Stack . Now i am trying to ship logs through collectd to my ubuntu machines and Openwrt routers . But i am unable to do so . This is my Server Logstash Conf. input { udp { port =\> 25826 host =\>… --- ## [\[WARN \]\[org.apache.kafka.clients.NetworkClient\] \[Consumer clientId=logstash-0, groupId=logstash\] Connection to node -1 could not be established. Broker may not be available](https://discuss.elastic.co/t/warn-org-apache-kafka-clients-networkclient-consumer-clientid-logstash-0-groupid-logstash-connection-to-node-1-could-not-be-established-broker-may-not-be-available/148571)
**Author:** [@aerys\_targaryen](https://discuss.elastic.co/u/aerys_targaryen)\ **Replies:** 0\ **Last updated:** [September 14, 2018, 7:29am UTC](https://discuss.elastic.co/t/warn-org-apache-kafka-clients-networkclient-consumer-clientid-logstash-0-groupid-logstash-connection-to-node-1-could-not-be-established-broker-may-not-be-available/148571 "2018-09-14T07:29:44Z")
Hello Im installing logstash arcsight module to ship logs from smartconnectors to es. my command looks like this: \[root@srv-hostname bin\] ./logstash --modules arcsight --setup --path.settings /etc/logstash/ " -M "… --- ## [Can we use multiline plugin with logstash input journald plugins?](https://discuss.elastic.co/t/can-we-use-multiline-plugin-with-logstash-input-journald-plugins/148089)
**Author:** [@talbehat](https://discuss.elastic.co/u/talbehat)\ **Replies:** 2\ **Last updated:** [September 14, 2018, 6:52am UTC](https://discuss.elastic.co/t/can-we-use-multiline-plugin-with-logstash-input-journald-plugins/148089 "2018-09-14T06:52:08Z")
If yes, what will be a way to use it with logstash input journald plugins. Please help me . --- ## [How many matches, we can have in grok filter - as example below](https://discuss.elastic.co/t/how-many-matches-we-can-have-in-grok-filter-as-example-below/148415)
**Author:** [@ramindia](https://discuss.elastic.co/u/ramindia)\ **Replies:** 4\ **Last updated:** [September 14, 2018, 6:14am UTC](https://discuss.elastic.co/t/how-many-matches-we-can-have-in-grok-filter-as-example-below/148415 "2018-09-14T06:14:59Z")
I have assorted logs coming to syslog file as one all.log, i am looking sort using different grok match, is this possible as below : match ? input { file { path =\> "/var/syslog-ng/raw/\*.log" start\_position =\> "beginn… --- ## [How to filter Filebeat input in Logstash from different log?](https://discuss.elastic.co/t/how-to-filter-filebeat-input-in-logstash-from-different-log/148480)
**Author:** [@Rahmat\_Agung\_W](https://discuss.elastic.co/u/Rahmat_Agung_W)\ **Replies:** 1\ **Last updated:** [September 14, 2018, 6:14am UTC](https://discuss.elastic.co/t/how-to-filter-filebeat-input-in-logstash-from-different-log/148480 "2018-09-14T06:14:06Z")
So, I have a VPS and have 2 json log from 2 different services. I used Filebeat to forward my log to ELK Stack Server. This is my filebeat.yml: filebeat.inputs: - type: log paths: - /home/cowrie/cowrie/log/cowrie.… --- ## [How to use the ruby{} filter to loop through k,v pairs after use the kv{} filter?](https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665)
**Author:** [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)\ **Replies:** 6\ **Last updated:** [September 14, 2018, 1:31am UTC](https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665 "2018-09-14T01:31:14Z")
After I use the kv plugin to parse the log, I want to filter the value further. What should I do? my data look like below: level="2" treatment="3" cmd1="\\"D:\\Program Files (x86)\\a\\safe\\modules\\setup.exe\\" /s /smartsile… --- ## [Logstash InfluxDB bad timestamp error](https://discuss.elastic.co/t/logstash-influxdb-bad-timestamp-error/148516)
**Author:** [@MrSantamaria](https://discuss.elastic.co/u/MrSantamaria)\ **Replies:** 0\ **Last updated:** [September 13, 2018, 9:24pm UTC](https://discuss.elastic.co/t/logstash-influxdb-bad-timestamp-error/148516 "2018-09-13T21:24:54Z")
Hello, I've almost got the configuration right but I am uncertain what is going on between the conversion of my data. I'm currently trying to turn mysql into influx. My input: is a jdbc, select \* from table query outpu… --- ## [Twitter Proxy Plugin Issue](https://discuss.elastic.co/t/twitter-proxy-plugin-issue/148512)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 0\ **Last updated:** [September 13, 2018, 8:32pm UTC](https://discuss.elastic.co/t/twitter-proxy-plugin-issue/148512 "2018-09-13T20:32:46Z")
Running Logstash 6.3.2 with Twitter Plugin version 3.0.8 on Windows Server 2012 R2, getting the below error when attempting to traverse a proxy. input { twitter { consumer\_key =\> "xxxx" consumer\_secret =\> "xxx… --- ## [Json validator in logstash](https://discuss.elastic.co/t/json-validator-in-logstash/148475)
**Author:** [@Malli](https://discuss.elastic.co/u/Malli)\ **Replies:** 1\ **Last updated:** [September 13, 2018, 5:46pm UTC](https://discuss.elastic.co/t/json-validator-in-logstash/148475 "2018-09-13T17:46:40Z")
I am using logstahs 2.3.4 and trying to parse thte json data. sometimes i am getting invalid json. I tried adding the skip\_on\_invalid\_json =\> 'true'. inside the json filter i am getting error and not giving any details.… --- ## [Hits.total in Elasticsearch filter plugin](https://discuss.elastic.co/t/hits-total-in-elasticsearch-filter-plugin/148342)
**Author:** [@javiyt](https://discuss.elastic.co/u/javiyt)\ **Replies:** 2\ **Last updated:** [September 13, 2018, 4:26pm UTC](https://discuss.elastic.co/t/hits-total-in-elasticsearch-filter-plugin/148342 "2018-09-13T16:26:01Z")
Hi everyone, I'm trying to enrich an index adding some geodata information, counting the neighbours according to a query. I'm using the elasticsearch filter to add this query, but I can't find a way to use either the \_… --- ## [Logstash does not start after upgrade from 6.1.2 to 6.4.0](https://discuss.elastic.co/t/logstash-does-not-start-after-upgrade-from-6-1-2-to-6-4-0/148478)
**Author:** [@t\_kauf](https://discuss.elastic.co/u/t_kauf)\ **Replies:** 0\ **Last updated:** [September 13, 2018, 4:04pm UTC](https://discuss.elastic.co/t/logstash-does-not-start-after-upgrade-from-6-1-2-to-6-4-0/148478 "2018-09-13T16:04:45Z")
Hi, today I upgraded my Elasticstack from Version 6.1.2 to 6.4.0. Since then Logstash does not start correctly anymore. I have pasted the logs / output here: https://paste.fedoraproject.org/paste/IR-6sdrXbmRgKgj5QNRbG… --- ## [Rabbitmq with logstash](https://discuss.elastic.co/t/rabbitmq-with-logstash/148449)
**Author:** [@khushboo1808aug](https://discuss.elastic.co/u/khushboo1808aug)\ **Replies:** 0\ **Last updated:** [September 13, 2018, 12:45pm UTC](https://discuss.elastic.co/t/rabbitmq-with-logstash/148449 "2018-09-13T12:45:29Z")
hi i tried connected with rabbit mq but i am not able to consume data from queue while my pipeline is open.Only at once data is inserted when i run config for connection. --- ## [Problème de pipeline](https://discuss.elastic.co/t/probleme-de-pipeline/148432)
**Author:** [@amerieme](https://discuss.elastic.co/u/amerieme)\ **Replies:** 1\ **Last updated:** [September 13, 2018, 12:25pm UTC](https://discuss.elastic.co/t/probleme-de-pipeline/148432 "2018-09-13T12:25:47Z")
Bonjour, J'ai un problème au niveau de pipeline Logstash à savoir : Problème lors de la déclaration des pipelines dans Kibana. Voici les versions de l’outil ELK que j’utilise : Outil Version Logstash 6.3.1 E… --- ## [JSON nested field loosing formating when referenced in add\_field](https://discuss.elastic.co/t/json-nested-field-loosing-formating-when-referenced-in-add-field/148126)
**Author:** [@marc1](https://discuss.elastic.co/u/marc1)\ **Replies:** 2\ **Last updated:** [September 13, 2018, 11:18am UTC](https://discuss.elastic.co/t/json-nested-field-loosing-formating-when-referenced-in-add-field/148126 "2018-09-13T11:18:08Z")
Hello, we have a use case where we would like to ingest specific JSON formatted messages into elasticsearch as well as convert the content of the input message into a new output JSON format and write it out to a Kafka to… --- ## [Parsing Akamai logs](https://discuss.elastic.co/t/parsing-akamai-logs/148311)
**Author:** [@Fabs](https://discuss.elastic.co/u/Fabs)\ **Replies:** 3\ **Last updated:** [September 13, 2018, 9:34am UTC](https://discuss.elastic.co/t/parsing-akamai-logs/148311 "2018-09-13T09:34:40Z")
Hi all, good morning I'm new about the ELK world. I have a problem on parsing Akamai logs. I report you a log as example: \< 2018-08-01 15:12:33 126.237.114.223 GET \[request\] 200 1270 0 "\[url\]" "Mozilla/5.0 (iPhone;… --- ## [Use the same connection for multiple Logstash configurations](https://discuss.elastic.co/t/use-the-same-connection-for-multiple-logstash-configurations/147736)
**Author:** [@Andre\_Silva](https://discuss.elastic.co/u/Andre_Silva)\ **Replies:** 6\ **Last updated:** [September 13, 2018, 9:10am UTC](https://discuss.elastic.co/t/use-the-same-connection-for-multiple-logstash-configurations/147736 "2018-09-13T09:10:19Z")
I'm using Logstash 2.4.1 to load data to Elasticsearch 2.4.6. I have the following Logstash config: input { jdbc { jdbc\_connection\_string =\> "jdbc:oracle:thin:@database:1521:db1" jdbc\_user =\> "user" jdbc\_p… --- ## [Delete those data that have been imported to ES](https://discuss.elastic.co/t/delete-those-data-that-have-been-imported-to-es/148317)
**Author:** [@ganessen23](https://discuss.elastic.co/u/ganessen23)\ **Replies:** 1\ **Last updated:** [September 13, 2018, 5:40am UTC](https://discuss.elastic.co/t/delete-those-data-that-have-been-imported-to-es/148317 "2018-09-13T05:40:39Z")
Hello, I am using ELK to import data from an external SQL database server (big data-millions of rows of data). I want to delete those data in the SQL server that have been imported to Elasticsearch. A prior solution w… --- ## [Logstash Hangs when filecount above 35K](https://discuss.elastic.co/t/logstash-hangs-when-filecount-above-35k/148403)
**Author:** [@caz](https://discuss.elastic.co/u/caz)\ **Replies:** 0\ **Last updated:** [September 13, 2018, 3:35am UTC](https://discuss.elastic.co/t/logstash-hangs-when-filecount-above-35k/148403 "2018-09-13T03:35:10Z")
Good day This is my first post so apologies if the standard is not followed. I ran into a strange problem when using logstash. I have noticed that my logstash seems to not process any files when the total count of fil… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=684) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=686)