# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=295 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 296 --- ## [Running multiple SQL Queries in logstash JDBC plugin](https://discuss.elastic.co/t/running-multiple-sql-queries-in-logstash-jdbc-plugin/251394)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 1\ **Last updated:** [October 8, 2020, 2:51pm UTC](https://discuss.elastic.co/t/running-multiple-sql-queries-in-logstash-jdbc-plugin/251394 "2020-10-08T14:51:57Z")
I know that multiple JDBC input plugins can be used in logstash to support the multiple SQL statements. Would like to know whether these are run sequentially or parallelly. Is there any way I can enforce them to run one … --- ## [Put array elements into new fields with logstash filter](https://discuss.elastic.co/t/put-array-elements-into-new-fields-with-logstash-filter/250980)
**Author:** [@LuigiDelavega](https://discuss.elastic.co/u/LuigiDelavega)\ **Replies:** 4\ **Last updated:** [October 8, 2020, 2:06pm UTC](https://discuss.elastic.co/t/put-array-elements-into-new-fields-with-logstash-filter/250980 "2020-10-08T14:06:56Z")
Hello, I have an issue finding a solution for my problem : My filebeat sends IIS logs to logstash, with an URIPATH field named iis.uri, it looks like : /aaaaa/bbbbb/cccccc/dddddd/eeeeee I copy this field i wanna keep… --- ## [Syslog data that is going into logstash not being picked up by elasticsearch](https://discuss.elastic.co/t/syslog-data-that-is-going-into-logstash-not-being-picked-up-by-elasticsearch/251332)
**Author:** [@tyrone](https://discuss.elastic.co/u/tyrone)\ **Replies:** 2\ **Last updated:** [October 8, 2020, 1:10pm UTC](https://discuss.elastic.co/t/syslog-data-that-is-going-into-logstash-not-being-picked-up-by-elasticsearch/251332 "2020-10-08T13:10:06Z")
I have all my servers pushing syslog data to 4 logstash servers, the logstash servers are not forwarding syslog information into elasticsearch, here is my conf file input { tcp { port =\> 5514 type =\> syslog } udp { … --- ## [Logstash filter doesn't apply on the output](https://discuss.elastic.co/t/logstash-filter-doesnt-apply-on-the-output/251443)
**Author:** [@BTH.S3](https://discuss.elastic.co/u/BTH.S3)\ **Replies:** 0\ **Last updated:** [October 8, 2020, 1:00pm UTC](https://discuss.elastic.co/t/logstash-filter-doesnt-apply-on-the-output/251443 "2020-10-08T13:00:29Z")
Recently I've deployed FileBeat in two remote servers for gathering log messages and send to the logstash on the local server. The connection between FireBeat and Logstash is perfect. The messages have been transferring… --- ## [Parse multiline JSON object from Docker log](https://discuss.elastic.co/t/parse-multiline-json-object-from-docker-log/251406)
**Author:** [@bellackn](https://discuss.elastic.co/u/bellackn)\ **Replies:** 1\ **Last updated:** [October 8, 2020, 12:45pm UTC](https://discuss.elastic.co/t/parse-multiline-json-object-from-docker-log/251406 "2020-10-08T12:45:11Z")
Hey there, I am trying to parse multiline JSON objects that Filebeat collects from Docker container JSON logfiles and then sends them to Logstash. Specifically, it's Java stacktraces. My output looks like this: {"log":… --- ## [Make Readable message in kibana](https://discuss.elastic.co/t/make-readable-message-in-kibana/251428)
**Author:** [@Bhavin\_Varsur](https://discuss.elastic.co/u/Bhavin_Varsur)\ **Replies:** 0\ **Last updated:** [October 8, 2020, 11:17am UTC](https://discuss.elastic.co/t/make-readable-message-in-kibana/251428 "2020-10-08T11:17:48Z")
In the above picture you can see message,timestamp and fields which is added by grok pattern. with use grok pattern take value from message and added into fields(set names in grok). now in fields you can see values b… --- ## [Unable to start logstash](https://discuss.elastic.co/t/unable-to-start-logstash/251396)
**Author:** [@tamilarasanbravo](https://discuss.elastic.co/u/tamilarasanbravo)\ **Replies:** 2\ **Last updated:** [October 8, 2020, 10:58am UTC](https://discuss.elastic.co/t/unable-to-start-logstash/251396 "2020-10-08T10:58:51Z")
Team, I am trying to start logstash but unable to do so as it says the port is being used but there is no such port used in the host. \[2020-10-08T04:18:10,305\]\[INFO \]\[logstash.inputs.http \]\[httpd\]\[c8cb23d9d60ec3c13… --- ## [Logstash create value based on duplicate field](https://discuss.elastic.co/t/logstash-create-value-based-on-duplicate-field/250852)
**Author:** [@bbwolf](https://discuss.elastic.co/u/bbwolf)\ **Replies:** 5\ **Last updated:** [October 8, 2020, 10:19am UTC](https://discuss.elastic.co/t/logstash-create-value-based-on-duplicate-field/250852 "2020-10-08T10:19:42Z")
Hello, I wan't know if it's possible to get duplicate value un field and create another field based on that value. Supposed I have logs: Field1, Field2, Field3 A,B, C A,D, E B, E, T B, R, H S, V, X What I wan't … --- ## [Logstash Fingerprint Issue](https://discuss.elastic.co/t/logstash-fingerprint-issue/251410)
**Author:** [@Pratik\_Sheth](https://discuss.elastic.co/u/Pratik_Sheth)\ **Replies:** 0\ **Last updated:** [October 8, 2020, 10:10am UTC](https://discuss.elastic.co/t/logstash-fingerprint-issue/251410 "2020-10-08T10:10:47Z")
Hi.. I have been having issue with the logstash configuration. I have set the configuration such that fingerprint of few fields is considered as doc\_id but The ID is not getting populated even tho hash are coming correc… --- ## [GREEDYDATA in Grok doesn't pick up a leading \[](https://discuss.elastic.co/t/greedydata-in-grok-doesnt-pick-up-a-leading/250999)
**Author:** [@joesheps](https://discuss.elastic.co/u/joesheps)\ **Replies:** 11\ **Last updated:** [October 8, 2020, 8:18am UTC](https://discuss.elastic.co/t/greedydata-in-grok-doesnt-pick-up-a-leading/250999 "2020-10-08T08:18:05Z")
I am receiving syslogs like this: Oct 5 13:37:13 10.20.30.40 %ASA-4-733100: \[ HTTP 80\] drop rate-1 exceeded. Current burst rate is 7 per second, max configured rate is 40; Current average rate is 23 per seco… --- ## [Getting errors after Logstash upgrade from 7.6.2 to 7.9.2](https://discuss.elastic.co/t/getting-errors-after-logstash-upgrade-from-7-6-2-to-7-9-2/251370)
**Author:** [@yummycake](https://discuss.elastic.co/u/yummycake)\ **Replies:** 2\ **Last updated:** [October 8, 2020, 7:33am UTC](https://discuss.elastic.co/t/getting-errors-after-logstash-upgrade-from-7-6-2-to-7-9-2/251370 "2020-10-08T07:33:11Z")
I recently upgraded my Logstash from 7.6.2 to the latest version 7.9.2. When I tested my configuration file, I received the output below. I did not get the error below output when I tested with version 7.6.2 May I know … --- ## [Logstash Grok pattern](https://discuss.elastic.co/t/logstash-grok-pattern/251373)
**Author:** [@Bhavin\_Varsur](https://discuss.elastic.co/u/Bhavin_Varsur)\ **Replies:** 4\ **Last updated:** [October 8, 2020, 7:25am UTC](https://discuss.elastic.co/t/logstash-grok-pattern/251373 "2020-10-08T07:25:28Z")
Grok pattern isn't working . before I used grok pattern and it creates a fields which was I added into pattern. but now it's not creating a fields Help me out of this Here is my grok pattern : grok { match… --- ## [Logstash daily indices write to wrong (yesterday) date](https://discuss.elastic.co/t/logstash-daily-indices-write-to-wrong-yesterday-date/251339)
**Author:** [@dorinand](https://discuss.elastic.co/u/dorinand)\ **Replies:** 0\ **Last updated:** [October 7, 2020, 8:44pm UTC](https://discuss.elastic.co/t/logstash-daily-indices-write-to-wrong-yesterday-date/251339 "2020-10-07T20:44:15Z")
I implement logstash elasticsearch output to create daily indices. The index name has this format: "default-%{\[kubernetes\]\[pod\]\[name\]}-%{+YYYY.MM.dd}" It works perfectly, it creates index, lets say for example for pod m… --- ## [Logstash memory use keeps increasing linearly over time](https://discuss.elastic.co/t/logstash-memory-use-keeps-increasing-linearly-over-time/250927)
**Author:** [@SHINRA](https://discuss.elastic.co/u/SHINRA)\ **Replies:** 6\ **Last updated:** [October 8, 2020, 4:43am UTC](https://discuss.elastic.co/t/logstash-memory-use-keeps-increasing-linearly-over-time/250927 "2020-10-08T04:43:04Z")
I have a logstash pipelines that runs tasks every day on a fixed time using the http input. Each task completes executing (from input of first pipeline to outputs of the last one) within few minutes. Rest of the day i… --- ## [Logstash can't create separate indexes](https://discuss.elastic.co/t/logstash-cant-create-separate-indexes/251270)
**Author:** [@Bhavin\_Varsur](https://discuss.elastic.co/u/Bhavin_Varsur)\ **Replies:** 2\ **Last updated:** [October 8, 2020, 3:31am UTC](https://discuss.elastic.co/t/logstash-cant-create-separate-indexes/251270 "2020-10-08T03:31:32Z")
I have two filebeat inputs which have tags and fields. In my pipeline.conf I used filter logs by theirs tags. but when the time of index create logstash takes index name as a %{\[fields\]\[log\_type\]}-2020-10-07. How can … --- ## [Logstash create and reference variables](https://discuss.elastic.co/t/logstash-create-and-reference-variables/251348)
**Author:** [@mohsin106](https://discuss.elastic.co/u/mohsin106)\ **Replies:** 0\ **Last updated:** [October 7, 2020, 10:28pm UTC](https://discuss.elastic.co/t/logstash-create-and-reference-variables/251348 "2020-10-07T22:28:41Z")
Hi, Is there a way to pull and field:value from one event, store it in some sort of variable and then add it to the next event? For example: event1: {device:"router1", in-rate:"1", out-rate:"2", interface:"eth01", des… --- ## [Aggregate Performing Unpredictably](https://discuss.elastic.co/t/aggregate-performing-unpredictably/251336)
**Author:** [@NomadicCodeGuy](https://discuss.elastic.co/u/NomadicCodeGuy)\ **Replies:** 2\ **Last updated:** [October 7, 2020, 8:05pm UTC](https://discuss.elastic.co/t/aggregate-performing-unpredictably/251336 "2020-10-07T20:05:21Z")
I am using the aggregate filter plugin to copy a field to all entries from the same file path that have a "nil" entry for that field. It works perfectly for around 95 percent of the time. The strange thing is that with e… --- ## [Dissect filter plugin to map nested fields](https://discuss.elastic.co/t/dissect-filter-plugin-to-map-nested-fields/251314)
**Author:** [@carlos-bernad](https://discuss.elastic.co/u/carlos-bernad)\ **Replies:** 2\ **Last updated:** [October 7, 2020, 5:01pm UTC](https://discuss.elastic.co/t/dissect-filter-plugin-to-map-nested-fields/251314 "2020-10-07T17:01:11Z")
I am listening to events like this with logstash: {"MYSQL":{"eqp":"ST\_22","eqpId":"TT\_1","label":"MONITOR : BW\_TE\_TRACKING \[20\]","local":"2020-09-23T22:35:29.855+05:00"},"tags":\["MYSQL"\]} My goal is to store it in Mong… --- ## [Timestamp field reordering events from patterns?](https://discuss.elastic.co/t/timestamp-field-reordering-events-from-patterns/251249)
**Author:** [@grumo35](https://discuss.elastic.co/u/grumo35)\ **Replies:** 4\ **Last updated:** [October 7, 2020, 3:33pm UTC](https://discuss.elastic.co/t/timestamp-field-reordering-events-from-patterns/251249 "2020-10-07T15:33:45Z")
Hi, Yesterday i cam across something i can't understand. Let's say i stopped my logstash at 10 AM and put it back on at 12, usually logs are coming and re-ordered as normal ( i can't figure how since @timestamp field s… --- ## [Extract only certain tags from an XML using XML filter plugin](https://discuss.elastic.co/t/extract-only-certain-tags-from-an-xml-using-xml-filter-plugin/251277)
**Author:** [@HariniGajendran](https://discuss.elastic.co/u/HariniGajendran)\ **Replies:** 2\ **Last updated:** [October 7, 2020, 3:21pm UTC](https://discuss.elastic.co/t/extract-only-certain-tags-from-an-xml-using-xml-filter-plugin/251277 "2020-10-07T15:21:05Z")
Hi, I am using XML filter to extract certain tags from an XML input. But the filter doesnt seem to be working correctly. PFB sample XML. \ **Author:** [@thdesy](https://discuss.elastic.co/u/thdesy)\ **Replies:** 0\ **Last updated:** [October 7, 2020, 2:01pm UTC](https://discuss.elastic.co/t/importing-ruby-class-fails-in-the-ruby-plugin/251281 "2020-10-07T14:01:36Z") Hi all, I am trying to use/offload some Ruby code in a filter to process XML events (due to various reasons, I have to move away from the XML input plugin). The problem is, that the import of 'nori' class fails within … --- ## [Join 2 indexes with 1 primary key logstash filter elasticsearch](https://discuss.elastic.co/t/join-2-indexes-with-1-primary-key-logstash-filter-elasticsearch/251279)
**Author:** [@charles97](https://discuss.elastic.co/u/charles97)\ **Replies:** 0\ **Last updated:** [October 7, 2020, 1:43pm UTC](https://discuss.elastic.co/t/join-2-indexes-with-1-primary-key-logstash-filter-elasticsearch/251279 "2020-10-07T13:43:09Z")
So tried to join a transfrom index to index table. I saw this as reference : How to join fields from multi events into single event? and it seems like it should work by using elasticsearch filter plugin. transform index… --- ## [Nested json inside a json also getting parsed into different fields using JSON Filter Plugin](https://discuss.elastic.co/t/nested-json-inside-a-json-also-getting-parsed-into-different-fields-using-json-filter-plugin/250939)
**Author:** [@ParashB](https://discuss.elastic.co/u/ParashB)\ **Replies:** 4\ **Last updated:** [October 7, 2020, 10:27am UTC](https://discuss.elastic.co/t/nested-json-inside-a-json-also-getting-parsed-into-different-fields-using-json-filter-plugin/250939 "2020-10-07T10:27:33Z")
Hi, I have a nested JSON inside a JSON field which is also getting parsed. I do not want to parse the nested JSON. example json: log : {"name":"abc","location":{"city":"New York","country":"US"}} Logstash filter: jso… --- ## [DHCP Logs to Logstash](https://discuss.elastic.co/t/dhcp-logs-to-logstash/250777)
**Author:** [@Craig2188](https://discuss.elastic.co/u/Craig2188)\ **Replies:** 8\ **Last updated:** [October 7, 2020, 10:03am UTC](https://discuss.elastic.co/t/dhcp-logs-to-logstash/250777 "2020-10-07T10:03:06Z")
I plan to use Filebeat to send the DHCP log file to Logstash. I have the following setup in Logstash to send the file to Elasticsearch: # Beats -\> Logstash -\> Elasticsearch pipeline. input { beats { port =\> 5044… --- ## [Logstash Aggregate copy last Event to Map](https://discuss.elastic.co/t/logstash-aggregate-copy-last-event-to-map/251149)
**Author:** [@siiman](https://discuss.elastic.co/u/siiman)\ **Replies:** 3\ **Last updated:** [October 7, 2020, 6:22am UTC](https://discuss.elastic.co/t/logstash-aggregate-copy-last-event-to-map/251149 "2020-10-07T06:22:12Z")
Hello there, I would like to combine several consecutive events. For this purpose I consider the use of the aggregation filter as useful. Since I don't have a specific start or end event, I will use example #3 in the Lo… --- ## [Benchmark jar is missing](https://discuss.elastic.co/t/benchmark-jar-is-missing/251207)
**Author:** [@Roman\_Kagan](https://discuss.elastic.co/u/Roman_Kagan)\ **Replies:** 0\ **Last updated:** [October 7, 2020, 4:11am UTC](https://discuss.elastic.co/t/benchmark-jar-is-missing/251207 "2020-10-07T04:11:48Z")
Hello: I wonder where can I find the benchmark.jar file in LogStash? Looks like it is missing from distribution. Blockquote #!/usr/bin/env bash java -cp "$(cd dirname $0/..; pwd)"'/tools/benchmark-cli/build/libs/be… --- ## [How to implement document\_id/doc\_as\_upsert/action setting with amazon\_es plugin](https://discuss.elastic.co/t/how-to-implement-document-id-doc-as-upsert-action-setting-with-amazon-es-plugin/251193)
**Author:** [@Rajesh\_Parui](https://discuss.elastic.co/u/Rajesh_Parui)\ **Replies:** 0\ **Last updated:** [October 6, 2020, 10:34pm UTC](https://discuss.elastic.co/t/how-to-implement-document-id-doc-as-upsert-action-setting-with-amazon-es-plugin/251193 "2020-10-06T22:34:13Z")
Hello, I am trying to upload the data to the elastic search (hosted in AWS) via logstash with the amazon\_es plugin as an output plugin. the issue i am facing with the amazon\_es plugin I am unable to implement the doc… --- ## [Filter HTTPD\_COMBINEDLOG does not display QS:agent field in Kibana](https://discuss.elastic.co/t/filter-httpd-combinedlog-does-not-display-qs-agent-field-in-kibana/250575)
**Author:** [@earlsanchez](https://discuss.elastic.co/u/earlsanchez)\ **Replies:** 1\ **Last updated:** [October 6, 2020, 9:18pm UTC](https://discuss.elastic.co/t/filter-httpd-combinedlog-does-not-display-qs-agent-field-in-kibana/250575 "2020-10-06T21:18:42Z")
Hi All, I'm new to ELK but seems this should be an easy filter. I'm using filebeat version 7.9.2 to send Apache access logs to logstash 7.9.2 and then on to AWS Elasticsearch and Kibana 7.7 with the AWS ES plug-in. He… --- ## [Aggregate events over a large period of time maintaining previous task maps](https://discuss.elastic.co/t/aggregate-events-over-a-large-period-of-time-maintaining-previous-task-maps/251178)
**Author:** [@arichiardi](https://discuss.elastic.co/u/arichiardi)\ **Replies:** 0\ **Last updated:** [October 6, 2020, 6:54pm UTC](https://discuss.elastic.co/t/aggregate-events-over-a-large-period-of-time-maintaining-previous-task-maps/251178 "2020-10-06T18:54:00Z")
Hi all, I am new to logstash and aggregate - but not too event sourcing/CQRS and I am looking for a strategy to handle the following problem. I have a product.created that contains the initial available products. I the… --- ## [Logstash filter conditions](https://discuss.elastic.co/t/logstash-filter-conditions/251175)
**Author:** [@MMerel](https://discuss.elastic.co/u/MMerel)\ **Replies:** 0\ **Last updated:** [October 6, 2020, 6:43pm UTC](https://discuss.elastic.co/t/logstash-filter-conditions/251175 "2020-10-06T18:43:38Z")
hi there, sorry to bother you but i´m completely new at this , I´m using version 7.9.2 of Elastic,Kibana ,Logstash and Beats , i successuflly connected them to a MQQT broker ang got some information about camera analytic… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=294) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=296)