# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=947 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 948 --- ## [Convert object/array datastructures to nested](https://discuss.elastic.co/t/convert-object-array-datastructures-to-nested/92594)
**Author:** [@jonasbareiss](https://discuss.elastic.co/u/jonasbareiss)\ **Replies:** 0\ **Last updated:** [July 11, 2017, 7:41am UTC](https://discuss.elastic.co/t/convert-object-array-datastructures-to-nested/92594 "2017-07-11T07:41:42Z")
Hi everyone, I have an elasticsearch index that was used as datastore, where only the source fields were used. Some data is in an object structure, but now I need to be able to aggregate on it. The array does not always… --- ## [5.4.3: plugin install problem](https://discuss.elastic.co/t/5-4-3-plugin-install-problem/92825)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 1\ **Last updated:** [July 12, 2017, 2:58pm UTC](https://discuss.elastic.co/t/5-4-3-plugin-install-problem/92825 "2017-07-12T14:58:32Z")
Hi, I'm trying to install the de\_dot plugin as our backend elasticsearch is 2.x and am getting an error: # bin/logstash-plugin install --no-verify de\_dot Installing de\_dot Plugin not found, aborting ERROR: Installation… --- ## [Removing whitespace from column in csv import](https://discuss.elastic.co/t/removing-whitespace-from-column-in-csv-import/92702)
**Author:** [@jay224](https://discuss.elastic.co/u/jay224)\ **Replies:** 3\ **Last updated:** [July 12, 2017, 2:10pm UTC](https://discuss.elastic.co/t/removing-whitespace-from-column-in-csv-import/92702 "2017-07-12T14:10:18Z")
I am trying to ingest csv file into ES. The csv file contains undetermined column names in header. I would to eliminate white space from column names. Since I don't know the column names I can't manually supply column na… --- ## [Get element from xml](https://discuss.elastic.co/t/get-element-from-xml/92632)
**Author:** [@mohammed\_cherif](https://discuss.elastic.co/u/mohammed_cherif)\ **Replies:** 4\ **Last updated:** [July 12, 2017, 2:10pm UTC](https://discuss.elastic.co/t/get-element-from-xml/92632 "2017-07-12T14:10:13Z")
I have the following logstash conf file: input { jdbc { jdbc\_connection\_string =\> "jdbc:oracle:thin:@host:1521:sid" jdbc\_user =\> "admin" jdbc\_password =\> "admin" jdbc\_validate\_connection =\>… --- ## [Filtering json fields to be outputted by logstash](https://discuss.elastic.co/t/filtering-json-fields-to-be-outputted-by-logstash/92253)
**Author:** [@aszac](https://discuss.elastic.co/u/aszac)\ **Replies:** 2\ **Last updated:** [July 12, 2017, 1:38pm UTC](https://discuss.elastic.co/t/filtering-json-fields-to-be-outputted-by-logstash/92253 "2017-07-12T13:38:17Z")
Hello! I am using logstash 5.4 to read a json interaction from kafka and then filter it so that: 1. interaction will be dropped unless field car.name exist 2. only fields car.name and car.model are outputted My confi… --- ## [About the "convert" function for CSVs](https://discuss.elastic.co/t/about-the-convert-function-for-csvs/92279)
**Author:** [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\ **Replies:** 5\ **Last updated:** [July 12, 2017, 1:24pm UTC](https://discuss.elastic.co/t/about-the-convert-function-for-csvs/92279 "2017-07-12T13:24:08Z")
Hey there ! I've got some questions about integrate CSVs into elasticsearch with Logstash. I've got a CSV with some datas like text and numbers (obviously), but also with IPv4 adresses and possibly Localisations (ca… --- ## [How to add a field?](https://discuss.elastic.co/t/how-to-add-a-field/92720)
**Author:** [@sweety](https://discuss.elastic.co/u/sweety)\ **Replies:** 1\ **Last updated:** [July 12, 2017, 12:34pm UTC](https://discuss.elastic.co/t/how-to-add-a-field/92720 "2017-07-12T12:34:26Z")
How to add a field in Logstash config file to know from which node I am receiving a log file? Thanks! --- ## [Determine when Logstash is done parsing](https://discuss.elastic.co/t/determine-when-logstash-is-done-parsing/92305)
**Author:** [@CDR](https://discuss.elastic.co/u/CDR)\ **Replies:** 5\ **Last updated:** [July 12, 2017, 12:33pm UTC](https://discuss.elastic.co/t/determine-when-logstash-is-done-parsing/92305 "2017-07-12T12:33:27Z")
I want to know when Logstash is done parsing. I know that information can be found in this link: But am unsure of where I put the code in the logstash.conf file? Also, in the comment states this: Unfortunately when … --- ## [Logstash sending same message to multiple indexes in ES](https://discuss.elastic.co/t/logstash-sending-same-message-to-multiple-indexes-in-es/92464)
**Author:** [@Zeal\_Vora](https://discuss.elastic.co/u/Zeal_Vora)\ **Replies:** 1\ **Last updated:** [July 12, 2017, 12:29pm UTC](https://discuss.elastic.co/t/logstash-sending-same-message-to-multiple-indexes-in-es/92464 "2017-07-12T12:29:35Z")
Hi We are using Logstash to send application logs to respective index in ElasticSearch. We have 2 applications which are sending logs to Logstash from filebeat. Each filebeat configuration for application has a resp… --- ## [Need to send mongoDb logs to some port then that can be used in logstash](https://discuss.elastic.co/t/need-to-send-mongodb-logs-to-some-port-then-that-can-be-used-in-logstash/92463)
**Author:** [@honeygidwani](https://discuss.elastic.co/u/honeygidwani)\ **Replies:** 1\ **Last updated:** [July 12, 2017, 12:28pm UTC](https://discuss.elastic.co/t/need-to-send-mongodb-logs-to-some-port-then-that-can-be-used-in-logstash/92463 "2017-07-12T12:28:05Z")
I am trying to send all mongoDb logs to some defined port This is because I am using replication and sharding in this. So i want to collect all the logs at some port . I am right now using file input for logstash to rea… --- ## [What filter should i use to remove the "\\" from my logfile](https://discuss.elastic.co/t/what-filter-should-i-use-to-remove-the-from-my-logfile/92017)
**Author:** [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\ **Replies:** 11\ **Last updated:** [July 12, 2017, 12:26pm UTC](https://discuss.elastic.co/t/what-filter-should-i-use-to-remove-the-from-my-logfile/92017 "2017-07-12T12:26:46Z")
this is an example of my log file.. {"LogMsg":"{\\"deviceId\\":\\"911490250554400\\",\\"description\\":\\"P701\\",\\"owner\\":\\"admin\\",\\"passcodeEnabled\\":\\"false\\",\\"batteryLevel\\":18.0,\\"internalAvailableMemory\\":2.78478848E8,\\… --- ## [Logstash syslog wont accept ports other than 514](https://discuss.elastic.co/t/logstash-syslog-wont-accept-ports-other-than-514/92445)
**Author:** [@X\_Calibur](https://discuss.elastic.co/u/X_Calibur)\ **Replies:** 1\ **Last updated:** [July 12, 2017, 12:26pm UTC](https://discuss.elastic.co/t/logstash-syslog-wont-accept-ports-other-than-514/92445 "2017-07-12T12:26:21Z")
I'm trying to get logstash to accept Syslog transmissions and it's giving me the following error messages: syslog listener died {:protocol=\>:udp, :address=\>"0.0.0.0:514", :exception=\>#\ **Author:** [@aditya](https://discuss.elastic.co/u/aditya)\ **Replies:** 1\ **Last updated:** [July 12, 2017, 12:24pm UTC](https://discuss.elastic.co/t/how-to-run-jdbc-mysql-importer-in-logstash-everytime/92425 "2017-07-12T12:24:02Z") How to run jdbc mysql importer in logstash whenever any transaction happen in mysql? --- ## [ELK with RabbitMQ](https://discuss.elastic.co/t/elk-with-rabbitmq/92481)
**Author:** [@ramesh.bahavathi](https://discuss.elastic.co/u/ramesh.bahavathi)\ **Replies:** 1\ **Last updated:** [July 12, 2017, 12:21pm UTC](https://discuss.elastic.co/t/elk-with-rabbitmq/92481 "2017-07-12T12:21:42Z")
Hi I have configured ELK stack, with Rabbit MQ. Log is shipping to RabbitMQ and but not moved into LogStash Indexer. Please assist. --- ## [How Logstash reads the same file name but with different content(keeps updating)?](https://discuss.elastic.co/t/how-logstash-reads-the-same-file-name-but-with-different-content-keeps-updating/89984)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 18\ **Last updated:** [July 12, 2017, 11:54am UTC](https://discuss.elastic.co/t/how-logstash-reads-the-same-file-name-but-with-different-content-keeps-updating/89984 "2017-07-12T11:54:40Z")
Hi, I am using logstash to read a log file. The file name reamins same but the content keeps changing for 1 hour(wipes off the previous logs and inserts new logs). But logstash is just reading last line of the file. He… --- ## [Illegal pattern component: t when matching date from filebeat system module](https://discuss.elastic.co/t/illegal-pattern-component-t-when-matching-date-from-filebeat-system-module/92042)
**Author:** [@lcorsini](https://discuss.elastic.co/u/lcorsini)\ **Replies:** 4\ **Last updated:** [July 12, 2017, 10:55am UTC](https://discuss.elastic.co/t/illegal-pattern-component-t-when-matching-date-from-filebeat-system-module/92042 "2017-07-12T10:55:07Z")
hi there, I'm using filebeat system module to send logfiles to elastic by using logstash. I've used the configurations listed here: but it's not working on the date filter giving me the Illegal pattern component:… --- ## [Mutate/remove fields with dot caracters](https://discuss.elastic.co/t/mutate-remove-fields-with-dot-caracters/91970)
**Author:** [@Romain\_Pelissier](https://discuss.elastic.co/u/Romain_Pelissier)\ **Replies:** 5\ **Last updated:** [July 12, 2017, 10:43am UTC](https://discuss.elastic.co/t/mutate-remove-fields-with-dot-caracters/91970 "2017-07-12T10:43:30Z")
Hi all! We use the input cef codec because we have some arcsight connectors that we want to send their events to our ELK platform. CEF is cool but we would like to remove a lot ot fields that are not necessary for the da… --- ## [Logstash trying to parse date to the wrong field](https://discuss.elastic.co/t/logstash-trying-to-parse-date-to-the-wrong-field/92121)
**Author:** [@marcogoldin](https://discuss.elastic.co/u/marcogoldin)\ **Replies:** 3\ **Last updated:** [July 12, 2017, 10:42am UTC](https://discuss.elastic.co/t/logstash-trying-to-parse-date-to-the-wrong-field/92121 "2017-07-12T10:42:10Z")
Hi, logstash suddenly shows a quite strange behaviour, i think it's trying to parse a text field into a date field. But i really don't know how to interpret this message in logstash-plain.log: \[2017-07-06T16:23:43,11… --- ## [Need to convert specific part of message to json in logstash](https://discuss.elastic.co/t/need-to-convert-specific-part-of-message-to-json-in-logstash/92088)
**Author:** [@honey](https://discuss.elastic.co/u/honey)\ **Replies:** 3\ **Last updated:** [July 12, 2017, 10:35am UTC](https://discuss.elastic.co/t/need-to-convert-specific-part-of-message-to-json-in-logstash/92088 "2017-07-12T10:35:52Z")
This is my message filed in logs {'v':'100001','lj':'f', 'level':'warnining', sys:{'sys':'',v:'',host:''},'tt':'a.db.all',o:{a:{b:{c:100}}}, err:{e:{status:300,message:'someException',stack:'abc prq ' } } } The outp… --- ## [Is it possible to mutate field in elastic using logstash?](https://discuss.elastic.co/t/is-it-possible-to-mutate-field-in-elastic-using-logstash/92051)
**Author:** [@bterhorst](https://discuss.elastic.co/u/bterhorst)\ **Replies:** 3\ **Last updated:** [July 12, 2017, 10:13am UTC](https://discuss.elastic.co/t/is-it-possible-to-mutate-field-in-elastic-using-logstash/92051 "2017-07-12T10:13:24Z")
Hi, Is it is possible to mutate a field (add/update) on an event already in elastic using logstash? I thought it would be possible to do that using the logstash-filter-elasticsearch but after reading the documentaion I… --- ## [Is it possible to use comma-separated-list-data as terms in Kibana?](https://discuss.elastic.co/t/is-it-possible-to-use-comma-separated-list-data-as-terms-in-kibana/92499)
**Author:** [@Igor123](https://discuss.elastic.co/u/Igor123)\ **Replies:** 5\ **Last updated:** [July 12, 2017, 9:18am UTC](https://discuss.elastic.co/t/is-it-possible-to-use-comma-separated-list-data-as-terms-in-kibana/92499 "2017-07-12T09:18:19Z")
Hi, I've tried searching, but can't find a way around what seems to be a rather basic usecase, so please send me to a suitable thread if there is one, but if not, here is my problem: I want to use the ELK stack to … --- ## [How to identify if logstash S3 Output plugin is pushing all data into s3 in batch?](https://discuss.elastic.co/t/how-to-identify-if-logstash-s3-output-plugin-is-pushing-all-data-into-s3-in-batch/92757)
**Author:** [@Abhishek\_Bande](https://discuss.elastic.co/u/Abhishek_Bande)\ **Replies:** 0\ **Last updated:** [July 12, 2017, 5:35am UTC](https://discuss.elastic.co/t/how-to-identify-if-logstash-s3-output-plugin-is-pushing-all-data-into-s3-in-batch/92757 "2017-07-12T05:35:47Z")
Hello, Anybody have idea on how we can identify that is my logstash is pushing output data to s3 in batch? --- ## [Need help with log filtering](https://discuss.elastic.co/t/need-help-with-log-filtering/92739)
**Author:** [@Leon1](https://discuss.elastic.co/u/Leon1)\ **Replies:** 1\ **Last updated:** [July 12, 2017, 1:05am UTC](https://discuss.elastic.co/t/need-help-with-log-filtering/92739 "2017-07-12T01:05:51Z")
Problem Description: I have the following log that needs to be filtered and I found that some of the Cisco ASA filter to be really useful. Jul 09 2017 23:34:50: %ASA-4-106023: Deny tcp src outside:1.2.3.4.5 ...... How… --- ## [Jdbc Oracle - only one row returned before closing](https://discuss.elastic.co/t/jdbc-oracle-only-one-row-returned-before-closing/92183)
**Author:** [@c95mbq](https://discuss.elastic.co/u/c95mbq)\ **Replies:** 1\ **Last updated:** [July 11, 2017, 10:40pm UTC](https://discuss.elastic.co/t/jdbc-oracle-only-one-row-returned-before-closing/92183 "2017-07-11T22:40:39Z")
Hi, I'm an absolute noob so apologizing in advance. I'm using version 5.4.0 of ELK and have a working Logstash configuration for indexing a csv file. This works ok but I wanted to try and query directly for data and ul… --- ## [JVM heapdump.bin Analyser](https://discuss.elastic.co/t/jvm-heapdump-bin-analyser/92732)
**Author:** [@rajkamalkool6](https://discuss.elastic.co/u/rajkamalkool6)\ **Replies:** 0\ **Last updated:** [July 11, 2017, 11:20pm UTC](https://discuss.elastic.co/t/jvm-heapdump-bin-analyser/92732 "2017-07-11T23:20:30Z")
Is there any filter or plugin available in Logstash 5.X to parse the heapdump.bin file generated from the server? --- ## [Logstash-codec-cloudtrail 5.5 throwing fatal key error](https://discuss.elastic.co/t/logstash-codec-cloudtrail-5-5-throwing-fatal-key-error/92727)
**Author:** [@zombyte](https://discuss.elastic.co/u/zombyte)\ **Replies:** 0\ **Last updated:** [July 11, 2017, 10:34pm UTC](https://discuss.elastic.co/t/logstash-codec-cloudtrail-5-5-throwing-fatal-key-error/92727 "2017-07-11T22:34:26Z")
Relatively new to elastic and logstash, but trying to get it to where cloudtrail stored in S3 can be sucked into Elastic and I'm trying to use the logstash-codec-cloudtrail to accomplish this. The plugin appears to work,… --- ## [Skip indexing document with null value](https://discuss.elastic.co/t/skip-indexing-document-with-null-value/92373)
**Author:** [@saisn](https://discuss.elastic.co/u/saisn)\ **Replies:** 1\ **Last updated:** [July 11, 2017, 8:19pm UTC](https://discuss.elastic.co/t/skip-indexing-document-with-null-value/92373 "2017-07-11T20:19:51Z")
Hi, I'm importing data from sql to elasticsearch using logstash. Is there a way in logstash to skip indexing document with null value. Thanks, Sai N --- ## [Grok filter and pattern in logstash](https://discuss.elastic.co/t/grok-filter-and-pattern-in-logstash/92404)
**Author:** [@0314e7ff87c297426a9e](https://discuss.elastic.co/u/0314e7ff87c297426a9e)\ **Replies:** 1\ **Last updated:** [July 11, 2017, 8:18pm UTC](https://discuss.elastic.co/t/grok-filter-and-pattern-in-logstash/92404 "2017-07-11T20:18:43Z")
I filtered the log using the grok pattern in logstash. The result logdata is too long. I want to disconnect from the loglevel unit, what should I do? --my logdata-- \[DEBUG\] \[2017-07-10 09:53:35,411\] pporan.maven.fram… --- ## [Loading a file having multiple XML elements, but document is not having ROOT element](https://discuss.elastic.co/t/loading-a-file-having-multiple-xml-elements-but-document-is-not-having-root-element/89423)
**Author:** [@bhattji007](https://discuss.elastic.co/u/bhattji007)\ **Replies:** 11\ **Last updated:** [July 11, 2017, 7:35pm UTC](https://discuss.elastic.co/t/loading-a-file-having-multiple-xml-elements-but-document-is-not-having-root-element/89423 "2017-07-11T19:35:07Z")
I am not able to find any post/documents/blogs/search to load details from XML file without root element. Here is my structure: rt-logs.gz file is having file containing daily real time messages, and each message have p… --- ## [Update existing document with fields from incoming document](https://discuss.elastic.co/t/update-existing-document-with-fields-from-incoming-document/92708)
**Author:** [@Mike\_Clarke](https://discuss.elastic.co/u/Mike_Clarke)\ **Replies:** 0\ **Last updated:** [July 11, 2017, 7:11pm UTC](https://discuss.elastic.co/t/update-existing-document-with-fields-from-incoming-document/92708 "2017-07-11T19:11:52Z")
I have documents with known unique IDs (document\_id) in Elasticsearch. I am using Filebeat to ship to Logstash and then to Elasticsearch. I would like to add fields from documents of a certain type coming from Filebeat… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=946) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=948)