# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=890 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 891 --- ## [KV trim\_key or remove\_char\_key](https://discuss.elastic.co/t/kv-trim-key-or-remove-char-key/104304)
**Author:** [@ctaroco](https://discuss.elastic.co/u/ctaroco)\ **Replies:** 1\ **Last updated:** [October 18, 2017, 5:19am UTC](https://discuss.elastic.co/t/kv-trim-key-or-remove-char-key/104304 "2017-10-18T05:19:42Z")
Hi, I'm receiving syslogs from different equipment with quite different formats, but every log always starts with something like this: \<188\>time=..... or \<177\>id=..... So, each time that I apply kv, I got a field name… --- ## [Increase log level for output plugin only](https://discuss.elastic.co/t/increase-log-level-for-output-plugin-only/104320)
**Author:** [@keithn](https://discuss.elastic.co/u/keithn)\ **Replies:** 1\ **Last updated:** [October 18, 2017, 5:14am UTC](https://discuss.elastic.co/t/increase-log-level-for-output-plugin-only/104320 "2017-10-18T05:14:44Z")
"Previously, you could only set a log level that applied to the entire Logstash product. Starting with 5.0, you can configure logging for a particular subsystem in Logstash." https://www.elastic.co/guide/en/logstash/curr… --- ## [Elastic search is not getting csv file data from logstash](https://discuss.elastic.co/t/elastic-search-is-not-getting-csv-file-data-from-logstash/104326)
**Author:** [@Navinder](https://discuss.elastic.co/u/Navinder)\ **Replies:** 0\ **Last updated:** [October 18, 2017, 5:03am UTC](https://discuss.elastic.co/t/elastic-search-is-not-getting-csv-file-data-from-logstash/104326 "2017-10-18T05:03:39Z")
Hi, Here's my logstash config file. input { file { path =\> "/home/app/estest7.csv" start\_position =\> "beginning" sincedb\_path =\> "/home/app/input.log" } } filter { csv { columns =\> \["user\_id","ts"\] separator =\> "… --- ## [Streaming transactional data from AWS RDS instance to Elasticsearch](https://discuss.elastic.co/t/streaming-transactional-data-from-aws-rds-instance-to-elasticsearch/103960)
**Author:** [@wsadiq](https://discuss.elastic.co/u/wsadiq)\ **Replies:** 3\ **Last updated:** [October 17, 2017, 8:16pm UTC](https://discuss.elastic.co/t/streaming-transactional-data-from-aws-rds-instance-to-elasticsearch/103960 "2017-10-17T20:16:18Z")
Hello, Our application keeps all its transactional data in an AWS RDS instance. I would like to provide business intelligence over this transactional data using Elasticsearch and Kibana. So my first task is to figure … --- ## [Logstash Plugin Install 6.x changed](https://discuss.elastic.co/t/logstash-plugin-install-6-x-changed/103969)
**Author:** [@Ricigliano](https://discuss.elastic.co/u/Ricigliano)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 8:11pm UTC](https://discuss.elastic.co/t/logstash-plugin-install-6-x-changed/103969 "2017-10-17T20:11:09Z")
Hi, I see instructions for installing the JDBC plug for logstash bin/plugin install logstash-input-jdbc but bin/plugin is no longer there, I see bin/Logstash-plugin Should I be using it? bin/logstash-plugin install… --- ## [Fields getting created for ever](https://discuss.elastic.co/t/fields-getting-created-for-ever/103956)
**Author:** [@cchooks2](https://discuss.elastic.co/u/cchooks2)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 8:10pm UTC](https://discuss.elastic.co/t/fields-getting-created-for-ever/103956 "2017-10-17T20:10:30Z")
2017-10-13T16:23:50.401-0400: 1056013.188: \[GC \[PSYoungGen: 785504K-\>64K(785920K)\] 898896K-\>113464K(3145216K), 0.0075690 secs\] \[Times: user=0.02 sys=0.00, real=0.00 secs\] %{TIMESTAMP\_ISO8601:event\_ts}: %{NUMBER:num:int}… --- ## [How to grok log file with different line formats](https://discuss.elastic.co/t/how-to-grok-log-file-with-different-line-formats/104289)
**Author:** [@amoghmudkavi](https://discuss.elastic.co/u/amoghmudkavi)\ **Replies:** 3\ **Last updated:** [October 17, 2017, 7:05pm UTC](https://discuss.elastic.co/t/how-to-grok-log-file-with-different-line-formats/104289 "2017-10-17T19:05:38Z")
my input log file looks like ... Infodom ABCDE RuleID 1481519674238 BuildFlag false InfoDate 20160331 TaskID Task25 Optional Params "$RUNID=1481925973714,$PHID=XXXX\_FR2052A\_COMMON\_RULES,$EXEID=1482870457650,$RUNSK=… --- ## [Simple Filter - Multiple OR](https://discuss.elastic.co/t/simple-filter-multiple-or/104295)
**Author:** [@jameswatson3](https://discuss.elastic.co/u/jameswatson3)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 7:04pm UTC](https://discuss.elastic.co/t/simple-filter-multiple-or/104295 "2017-10-17T19:04:08Z")
I have a working filter that is meeting my needs but I'm wondering about the efficiency as it grows. My apologies for the formatting as I'm inexperienced on this forum but my filter is as follows: filter { if \[type\] ==… --- ## [Logstash CSV output to Kibana](https://discuss.elastic.co/t/logstash-csv-output-to-kibana/103566)
**Author:** [@bobwilly](https://discuss.elastic.co/u/bobwilly)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 6:58pm UTC](https://discuss.elastic.co/t/logstash-csv-output-to-kibana/103566 "2017-10-17T18:58:04Z")
im trying to get my .CSV file in Kibana for visualisation. It feels like Im close to get it work but I can\`t figer out how to get my output right. In Kibana I see my .csv file as: message: News,test@email.com,10.10.10.… --- ## [Write filter for multiline in logstash filter](https://discuss.elastic.co/t/write-filter-for-multiline-in-logstash-filter/103888)
**Author:** [@tharu85](https://discuss.elastic.co/u/tharu85)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 6:44pm UTC](https://discuss.elastic.co/t/write-filter-for-multiline-in-logstash-filter/103888 "2017-10-17T18:44:14Z")
I need to write a logstash filter for linux multiline log using GROK. My log sample is like as below. 2017-05-22T04:28:12.444+05:30 WARN \[BlockingBatchedESOutput\] Error while waiting for healthy Elasticsearch cluster. … --- ## [Input file plugin doesn't process multiple input files on intensive logging in any of them](https://discuss.elastic.co/t/input-file-plugin-doesnt-process-multiple-input-files-on-intensive-logging-in-any-of-them/103871)
**Author:** [@vkakhnych](https://discuss.elastic.co/u/vkakhnych)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 6:43pm UTC](https://discuss.elastic.co/t/input-file-plugin-doesnt-process-multiple-input-files-on-intensive-logging-in-any-of-them/103871 "2017-10-17T18:43:44Z")
Tested on logstash 2.3, centos6. Simply logstash config: input { file { path =\> \[ "/tmp/\*-running.log" \] } } output { stdout { codec =\> json\_lines } } And simply logs popolate script: #!/bin/sh while t… --- ## [Removing fields from Twitter pipeline via conf file](https://discuss.elastic.co/t/removing-fields-from-twitter-pipeline-via-conf-file/103976)
**Author:** [@sharon1](https://discuss.elastic.co/u/sharon1)\ **Replies:** 2\ **Last updated:** [October 17, 2017, 6:00pm UTC](https://discuss.elastic.co/t/removing-fields-from-twitter-pipeline-via-conf-file/103976 "2017-10-17T18:00:14Z")
Good day to you all, Recently I installed ES on my laptop and I have been playing around with it. At the moment I have got a Twitter pipeline running. All Twitter fields are being loaded into ES. I would like to remov… --- ## [Logstash upsert partial document](https://discuss.elastic.co/t/logstash-upsert-partial-document/102249)
**Author:** [@Justin\_Cross](https://discuss.elastic.co/u/Justin_Cross)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 5:45pm UTC](https://discuss.elastic.co/t/logstash-upsert-partial-document/102249 "2017-10-17T17:45:28Z")
I have an index that contains some documents where field that are null and eventually once another particular row comes that has data for these fields, I want to upsert all old documents containing a particular document … --- ## [Identify filebeat input with tags and input type](https://discuss.elastic.co/t/identify-filebeat-input-with-tags-and-input-type/102288)
**Author:** [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 5:44pm UTC](https://discuss.elastic.co/t/identify-filebeat-input-with-tags-and-input-type/102288 "2017-10-17T17:44:04Z")
HI, I am unsure of how to identify filebeat input both with tags and type input { beats { client\_inactivity\_timeout =\> 86400 port =\> 5044 type =\> "log" } } filter { if "vus" in \[tags\] {… --- ## [Packetbeat plugin is not working](https://discuss.elastic.co/t/packetbeat-plugin-is-not-working/102355)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 5:41pm UTC](https://discuss.elastic.co/t/packetbeat-plugin-is-not-working/102355 "2017-10-17T17:41:13Z")
Hi Guys, Somehow my packetbeat plugin is not working which is installed on my DNS server and is forced to capture DNS packets How Can I troubleshoot further? packetbeat.interfaces.device: eth0 packetbeat.flows: Set … --- ## [Logstash 5.6.2 - Pipeline aborted - Unrecognized SSL message](https://discuss.elastic.co/t/logstash-5-6-2-pipeline-aborted-unrecognized-ssl-message/102390)
**Author:** [@west415](https://discuss.elastic.co/u/west415)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 5:38pm UTC](https://discuss.elastic.co/t/logstash-5-6-2-pipeline-aborted-unrecognized-ssl-message/102390 "2017-10-17T17:38:23Z")
Hi, I'm running Logstash 5.6.2 on Ubuntu 16.x and getting this error when I try to run my config using (logstash -f myconfigfile.conf) My input part of the config file looks like this: input { jdbc { jdbc… --- ## [Intermittent ArrayIndexOutOfBoundsException in dissect](https://discuss.elastic.co/t/intermittent-arrayindexoutofboundsexception-in-dissect/104095)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 5:03pm UTC](https://discuss.elastic.co/t/intermittent-arrayindexoutofboundsexception-in-dissect/104095 "2017-10-17T17:03:32Z")
I am trying to load UK housing price data in a CSV from HM land registry. Column 4 in that CSV should be the postcode, but sometimes it is empty, and sometimes that causes dissect to throw an ArrayIndexOutOfBoundsExcepti… --- ## [Logstash in Docker - Combine 2 events into 1 event](https://discuss.elastic.co/t/logstash-in-docker-combine-2-events-into-1-event/104278)
**Author:** [@mn\_dev](https://discuss.elastic.co/u/mn_dev)\ **Replies:** 0\ **Last updated:** [October 17, 2017, 4:23pm UTC](https://discuss.elastic.co/t/logstash-in-docker-combine-2-events-into-1-event/104278 "2017-10-17T16:23:12Z")
I am running ElasticStack in Docker via their official images; however, I'm currently receiving the following error message when I attempt to use the Logstash - aggregate plugin to combine events that have the same Reque… --- ## [Kafka input - offset committing to frequently](https://discuss.elastic.co/t/kafka-input-offset-committing-to-frequently/104262)
**Author:** [@blacklobo](https://discuss.elastic.co/u/blacklobo)\ **Replies:** 0\ **Last updated:** [October 17, 2017, 2:29pm UTC](https://discuss.elastic.co/t/kafka-input-offset-committing-to-frequently/104262 "2017-10-17T14:29:27Z")
I have a Kafka cluster that I am reading data from. However I am seeing offsets being committed to the cluster much more frequently than what I have in my settings. Additionally I am seeing offsets being written at ver… --- ## [Convert json to xml](https://discuss.elastic.co/t/convert-json-to-xml/103863)
**Author:** [@rm1890](https://discuss.elastic.co/u/rm1890)\ **Replies:** 16\ **Last updated:** [October 17, 2017, 11:56am UTC](https://discuss.elastic.co/t/convert-json-to-xml/103863 "2017-10-17T11:56:01Z")
hi all, I wonder if there is a way to convert json to xml. So, in my json data one of the fields has string type. It is actually xml content but represented as string in double quotes. Is there any best practice to conv… --- ## [Output to Elastic sub index](https://discuss.elastic.co/t/output-to-elastic-sub-index/104129)
**Author:** [@3for1](https://discuss.elastic.co/u/3for1)\ **Replies:** 3\ **Last updated:** [October 17, 2017, 11:18am UTC](https://discuss.elastic.co/t/output-to-elastic-sub-index/104129 "2017-10-17T11:18:47Z")
How should I configure the logstash output so that the data gets into a sub index. E.g: index =\> "/food/%{food\_type}" where food\_type is a field I have created in filter {} --- ## [More Logstash threads?](https://discuss.elastic.co/t/more-logstash-threads/104084)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 3\ **Last updated:** [October 17, 2017, 11:17am UTC](https://discuss.elastic.co/t/more-logstash-threads/104084 "2017-10-17T11:17:07Z")
HI, our Logstash does a lot of calculations and is getting stuck under a lot of info. The question is how to fix this, one way we saw was to raise the Xms and Xmx. Is there a way to give the logstash to work with more t… --- ## [Logstash\_Dns timeout issue](https://discuss.elastic.co/t/logstash-dns-timeout-issue/104220)
**Author:** [@Raj\_Kumar](https://discuss.elastic.co/u/Raj_Kumar)\ **Replies:** 0\ **Last updated:** [October 17, 2017, 9:46am UTC](https://discuss.elastic.co/t/logstash-dns-timeout-issue/104220 "2017-10-17T09:46:24Z")
Hi All, Is there any way to fix this issue every once in 15 days am getting this message \[2017-10-17T11:36:35,433\]\[ERROR\]\[logstash.filters.dns \] DNS: timeout on resolving address. {:field=\>"hostnameDst", :value=\>"… --- ## [Logstash on Windows with CSVs - Weird Behavior](https://discuss.elastic.co/t/logstash-on-windows-with-csvs-weird-behavior/104052)
**Author:** [@ahaw021](https://discuss.elastic.co/u/ahaw021)\ **Replies:** 2\ **Last updated:** [October 17, 2017, 7:37am UTC](https://discuss.elastic.co/t/logstash-on-windows-with-csvs-weird-behavior/104052 "2017-10-17T07:37:26Z")
Hi All I hope I am having a 1d10t error but thought I would ask anyway. I have downloaded a bunch of CSV files which sit on a folder (in my case: E:\\GITHUB\\SPLUNK-ELK-CERTIFICATE-TRANSPARENCY\\playground\\NEWBIE) \[image… --- ## [Logstash issue with Netflow module](https://discuss.elastic.co/t/logstash-issue-with-netflow-module/102898)
**Author:** [@juanquy](https://discuss.elastic.co/u/juanquy)\ **Replies:** 35\ **Last updated:** [October 17, 2017, 7:04am UTC](https://discuss.elastic.co/t/logstash-issue-with-netflow-module/102898 "2017-10-17T07:04:08Z")
HI : :smile: I'm having this issue with logstash with netflow module active: If I ran it from CLI with this command : ./logstash --modules netflow It work perfect , Kibana get the index patter netflow-\* and I can se… --- ## [Not able to connect to hive with logstash input](https://discuss.elastic.co/t/not-able-to-connect-to-hive-with-logstash-input/104046)
**Author:** [@Navinder](https://discuss.elastic.co/u/Navinder)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 6:28am UTC](https://discuss.elastic.co/t/not-able-to-connect-to-hive-with-logstash-input/104046 "2017-10-17T06:28:20Z")
Hi I am using below input in logstash. input { jdbc { jdbc\_driver\_library =\> "home/app/hive-jdbc-2.0.0.jar" jdbc\_driver\_class =\> "org.apache.hive.jdbc.HiveDriver" jdbc\_connection\_string =\> "jdbc:hive2://xxxxxxx:10000/d… --- ## [Logstash jdbc ignore default last run folder](https://discuss.elastic.co/t/logstash-jdbc-ignore-default-last-run-folder/104180)
**Author:** [@uLan](https://discuss.elastic.co/u/uLan)\ **Replies:** 0\ **Last updated:** [October 17, 2017, 5:53am UTC](https://discuss.elastic.co/t/logstash-jdbc-ignore-default-last-run-folder/104180 "2017-10-17T05:53:46Z")
I am using the logstash jdbc plugin and everything is working fine, but it gives out an error every time the sql query is executed. The error is: \[Screenshot from 2017-10-17 13-50-57\] This causes no problem what so e… --- ## [KV filter is unable to handle = within a key value](https://discuss.elastic.co/t/kv-filter-is-unable-to-handle-within-a-key-value/103878)
**Author:** [@elvarb](https://discuss.elastic.co/u/elvarb)\ **Replies:** 5\ **Last updated:** [October 17, 2017, 5:34am UTC](https://discuss.elastic.co/t/kv-filter-is-unable-to-handle-within-a-key-value/103878 "2017-10-17T05:34:40Z")
When I have the KV filter run against this string last="3367" post="" ltime="3" Instead of ending up with last: 3367 post: " ltime="3 I end up with last: 3367 post: \\ ltime: \\"3" I am expecting the KV filter to acc… --- ## [Looking for a way to "detect json"](https://discuss.elastic.co/t/looking-for-a-way-to-detect-json/102263)
**Author:** [@Napsty](https://discuss.elastic.co/u/Napsty)\ **Replies:** 5\ **Last updated:** [October 17, 2017, 5:45am UTC](https://discuss.elastic.co/t/looking-for-a-way-to-detect-json/102263 "2017-10-17T05:45:45Z")
On my Docker servers I use the GELF log plugin to write the logs in GELF format to Logstash. On the Logstash side I prepared the following listener: input { gelf { host =\> "0.0.0.0" port =\> 5000 type =\> "… --- ## [Help with a grok filter for Docker](https://discuss.elastic.co/t/help-with-a-grok-filter-for-docker/103932)
**Author:** [@DannielWhatever](https://discuss.elastic.co/u/DannielWhatever)\ **Replies:** 1\ **Last updated:** [October 17, 2017, 5:34am UTC](https://discuss.elastic.co/t/help-with-a-grok-filter-for-docker/103932 "2017-10-17T05:34:07Z")
Hello guys, I pretty noob in Elastic, and I trying to implement a filter, for the messages of a SpringBoot application running on docker, that is sending the logs to Logstash using the syslog driver. The messages seems… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=889) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=891)