# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1063 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1064 --- ## [Grok filter if \[type\]](https://discuss.elastic.co/t/grok-filter-if-type/69430)
**Author:** [@ariceELK](https://discuss.elastic.co/u/ariceELK)\ **Replies:** 4\ **Last updated:** [December 19, 2016, 12:40pm UTC](https://discuss.elastic.co/t/grok-filter-if-type/69430 "2016-12-19T12:40:01Z")
Hi, Were using logstash 5.1.1 adn cannot get the if statement to work in the the filter: filter { if \[type\] == "log" { grok { patterns\_dir =\> \["/opt/grok/patterns"\] match =\> { "message" =\> \[ Grok … --- ## [Logstash-filter-aggregate (2.5.0) fails with latest logstash (5.0.2)](https://discuss.elastic.co/t/logstash-filter-aggregate-2-5-0-fails-with-latest-logstash-5-0-2/68269)
**Author:** [@jeslopalo](https://discuss.elastic.co/u/jeslopalo)\ **Replies:** 2\ **Last updated:** [December 19, 2016, 11:48am UTC](https://discuss.elastic.co/t/logstash-filter-aggregate-2-5-0-fails-with-latest-logstash-5-0-2/68269 "2016-12-19T11:48:05Z")
Hi, after update logstash from 5.0.0 to 5.0.2 my pipeline fails when aggregating events with logstash-filter-aggregate. Is something changed with aggragate syntax? It's a compatibility issue? Sending Logstash's … --- ## [RabbitMQ to LogStash / Elasticsearch](https://discuss.elastic.co/t/rabbitmq-to-logstash-elasticsearch/40676)
**Author:** [@Akshay\_Shah](https://discuss.elastic.co/u/Akshay_Shah)\ **Replies:** 4\ **Last updated:** [December 19, 2016, 8:59am UTC](https://discuss.elastic.co/t/rabbitmq-to-logstash-elasticsearch/40676 "2016-12-19T08:59:21Z")
I m very new to ElasticSearch and LogStash. and I want to move messages out from RabbitMQ for ElasticSearch and further Kibana to visualize aggregations. Application Messages \>\> RabbitMQ \>\> LogStash \>\> Index in Elasti… --- ## [Output logs with date of indexing Logstash](https://discuss.elastic.co/t/output-logs-with-date-of-indexing-logstash/69000)
**Author:** [@lifesaver13](https://discuss.elastic.co/u/lifesaver13)\ **Replies:** 0\ **Last updated:** [December 14, 2016, 10:12am UTC](https://discuss.elastic.co/t/output-logs-with-date-of-indexing-logstash/69000 "2016-12-14T10:12:27Z")
Hey everbody, I'am moving olds Elasticsearch logs (2.2.0) to a higer version (5.0.2) with Logstash. It's working but I can't index the logs in the new version with the date of indexing of the old logs. I'm using the … --- ## [Logstash support to AIX?](https://discuss.elastic.co/t/logstash-support-to-aix/69400)
**Author:** [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Replies:** 1\ **Last updated:** [December 19, 2016, 7:31am UTC](https://discuss.elastic.co/t/logstash-support-to-aix/69400 "2016-12-19T07:31:05Z")
Hi, Does Logstash supports to AIX? We have some ESB logs on AIX machine, those we want to ship to some remote RHEL server in files, OR to logstash instance running on RHEL machine. Now we have 3 options. 1) LSF (AIX vers… --- ## [Can't get the value from mysql-slow.log](https://discuss.elastic.co/t/cant-get-the-value-from-mysql-slow-log/69126)
**Author:** [@houfan](https://discuss.elastic.co/u/houfan)\ **Replies:** 1\ **Last updated:** [December 19, 2016, 1:42am UTC](https://discuss.elastic.co/t/cant-get-the-value-from-mysql-slow-log/69126 "2016-12-19T01:42:25Z")
input : # Query\_time: 0.000148 Lock\_time: 0.000023 Rows\_sent: 0 Rows\_examined: 202 grok pattern is : Query\_time: %{NUMBER:query\_time:float}\\s+Lock\_time: %{NUMBER:lock\_time:float}\\s+Rows\_sent: %{NUMBER:rows\_sent:in… --- ## [Too many open threads causes other processes to crash](https://discuss.elastic.co/t/too-many-open-threads-causes-other-processes-to-crash/69157)
**Author:** [@gcsaba2](https://discuss.elastic.co/u/gcsaba2)\ **Replies:** 1\ **Last updated:** [December 19, 2016, 1:20am UTC](https://discuss.elastic.co/t/too-many-open-threads-causes-other-processes-to-crash/69157 "2016-12-19T01:20:13Z")
Hello, We had an issue where logstash opened over 5000 threads. As a consequence, the OS didn't let other processes start on the machine and we had to kill the logstash process. I'm guessing that logstash had one thr… --- ## [Jdbc sql\_last\_value](https://discuss.elastic.co/t/jdbc-sql-last-value/69156)
**Author:** [@gilisade](https://discuss.elastic.co/u/gilisade)\ **Replies:** 1\ **Last updated:** [December 18, 2016, 9:04pm UTC](https://discuss.elastic.co/t/jdbc-sql-last-value/69156 "2016-12-18T21:04:53Z")
Hi i have a few tables in MSSQL and ORACLE DB that i want to insert to elasticsearch i am using logstash for that with the jdbc input i want the statement to run every 3 minutes now i dont know how to "tell" logstash to… --- ## [Aggregation enrichment in Logstash with Elasticsearch filter](https://discuss.elastic.co/t/aggregation-enrichment-in-logstash-with-elasticsearch-filter/69308)
**Author:** [@cscsabai](https://discuss.elastic.co/u/cscsabai)\ **Replies:** 1\ **Last updated:** [December 18, 2016, 8:57pm UTC](https://discuss.elastic.co/t/aggregation-enrichment-in-logstash-with-elasticsearch-filter/69308 "2016-12-18T20:57:54Z")
Hi, I'd like to make an aggregated value enrichment every time when new log entry generated. This is a performance value and I'd like to expand it with "average of this value in the past 4 weeks in this time period" valu… --- ## [Add field from JSON / logstash filter](https://discuss.elastic.co/t/add-field-from-json-logstash-filter/69364)
**Author:** [@Makra](https://discuss.elastic.co/u/Makra)\ **Replies:** 3\ **Last updated:** [December 18, 2016, 6:26pm UTC](https://discuss.elastic.co/t/add-field-from-json-logstash-filter/69364 "2016-12-18T18:26:02Z")
Hi I am trying to add a filed from the parsed JSON, but the logstash filter does not add the value of the filed, instead it just add the whole string from the add field section. test.conf input { stdin {… --- ## [Using filter on nested objects](https://discuss.elastic.co/t/using-filter-on-nested-objects/68966)
**Author:** [@shambar](https://discuss.elastic.co/u/shambar)\ **Replies:** 1\ **Last updated:** [December 18, 2016, 4:12pm UTC](https://discuss.elastic.co/t/using-filter-on-nested-objects/68966 "2016-12-18T16:12:06Z")
I am having an issue trying to get logstash to see the items in the array beyond the first item. the data will come in like. {"id": "430229930", "item\_tbl" :\[{"itemid": 1,"itemname":"eggs","quantity": 10},{"itemid"… --- ## [How to add new gem for my plugin usage?](https://discuss.elastic.co/t/how-to-add-new-gem-for-my-plugin-usage/68985)
**Author:** [@itzikHan](https://discuss.elastic.co/u/itzikHan)\ **Replies:** 1\ **Last updated:** [December 18, 2016, 4:11pm UTC](https://discuss.elastic.co/t/how-to-add-new-gem-for-my-plugin-usage/68985 "2016-12-18T16:11:03Z")
I want to add a new gem (win32ole) for a plugin i'm building. After i installed the gem, what should i do in order for LogStash to recognize this gem from my plugin file? --- ## [Custom Index Name - Logstash](https://discuss.elastic.co/t/custom-index-name-logstash/44222)
**Author:** [@Sameer\_Dharur](https://discuss.elastic.co/u/Sameer_Dharur)\ **Replies:** 7\ **Last updated:** [December 18, 2016, 1:48pm UTC](https://discuss.elastic.co/t/custom-index-name-logstash/44222 "2016-12-18T13:48:47Z")
Hello! I see that logstash is creating a new index on the Elasticsearch server everyday with the name 'logstash-yyyy.mm.dd' However, I don't want a new file like this everyday and just want to give a custom name for… --- ## [How to reparse some files](https://discuss.elastic.co/t/how-to-reparse-some-files/69185)
**Author:** [@valugi](https://discuss.elastic.co/u/valugi)\ **Replies:** 4\ **Last updated:** [December 18, 2016, 12:03pm UTC](https://discuss.elastic.co/t/how-to-reparse-some-files/69185 "2016-12-18T12:03:00Z")
Hi, I am in a development/testing phase of implementing the ELK stack. So I play and modify patterns and would like to see in Kibana the result of those changes as soon as possible. On my system, I am trying to test a m… --- ## [DNS Filter - only first value in reverse array works](https://discuss.elastic.co/t/dns-filter-only-first-value-in-reverse-array-works/69301)
**Author:** [@sliddjur](https://discuss.elastic.co/u/sliddjur)\ **Replies:** 2\ **Last updated:** [December 17, 2016, 10:10pm UTC](https://discuss.elastic.co/t/dns-filter-only-first-value-in-reverse-array-works/69301 "2016-12-17T22:10:41Z")
This is my filter: filter { if \[type\] == "netflow" { mutate { add\_field =\> { "\[netflow\]\[ipv4\_dst\_host\]" =\> "%{\[netflow\]\[ipv4\_dst\_addr\]}" … --- ## [Logging additional details](https://discuss.elastic.co/t/logging-additional-details/69337)
**Author:** [@shivabharat](https://discuss.elastic.co/u/shivabharat)\ **Replies:** 1\ **Last updated:** [December 17, 2016, 8:39am UTC](https://discuss.elastic.co/t/logging-additional-details/69337 "2016-12-17T08:39:54Z")
I would like to know if its possible to gather additional details in logstash at the time of filtering or logging. For e.g in the filter section if we see a particular error code , can we gather system related details li… --- ## [How to call a command from within a mutate email block then include the output in the email?](https://discuss.elastic.co/t/how-to-call-a-command-from-within-a-mutate-email-block-then-include-the-output-in-the-email/69066)
**Author:** [@TomK](https://discuss.elastic.co/u/TomK)\ **Replies:** 7\ **Last updated:** [December 17, 2016, 3:02am UTC](https://discuss.elastic.co/t/how-to-call-a-command-from-within-a-mutate-email-block-then-include-the-output-in-the-email/69066 "2016-12-17T03:02:05Z")
(Rephrasing) We're trying to achieve the following: 1) Retrieve a UID from a log file entry / line. 2) Lookup that UID to get the real User Name. 3) Email the Log Entry / Line , the UID and the User Name (based on th… --- ## [Logstash/Elasticsearch process overwriting doc version each time?](https://discuss.elastic.co/t/logstash-elasticsearch-process-overwriting-doc-version-each-time/69317)
**Author:** [@west415](https://discuss.elastic.co/u/west415)\ **Replies:** 1\ **Last updated:** [December 17, 2016, 12:09am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-process-overwriting-doc-version-each-time/69317 "2016-12-17T00:09:37Z")
HI, I'm very new to the elastic world, but I was able to get logstash setup and successfully moving data from a SQL Server into my index in elasticsearch using the JDBC driver setup. As of now, my ETL process is curr… --- ## [Work with Redis in Logstash 1.5 and 5.1](https://discuss.elastic.co/t/work-with-redis-in-logstash-1-5-and-5-1/69311)
**Author:** [@dshmatov](https://discuss.elastic.co/u/dshmatov)\ **Replies:** 0\ **Last updated:** [December 16, 2016, 6:02pm UTC](https://discuss.elastic.co/t/work-with-redis-in-logstash-1-5-and-5-1/69311 "2016-12-16T18:02:05Z")
I have curent logging flow: Filebeat -\> Redis -\> Logstash -\> Elasticsearch At first time when I used Logstash 5.1 and Redis I found strange behavior in Logstash messages collecting from Redis. The message queue is jump… --- ## [Logstash won't create ES indexes (but will store data)](https://discuss.elastic.co/t/logstash-wont-create-es-indexes-but-will-store-data/69193)
**Author:** [@gamaier](https://discuss.elastic.co/u/gamaier)\ **Replies:** 2\ **Last updated:** [December 16, 2016, 5:43pm UTC](https://discuss.elastic.co/t/logstash-wont-create-es-indexes-but-will-store-data/69193 "2016-12-16T17:43:21Z")
I'm having an odd issue with logstash that I've never run into before. I spun up a new 5.1 stack the other day and for some reason logstash doesn't seem to be creating an index by itself. I'll get the error: :respons… --- ## [Ability to remove tags field from output broke in 5.0.1 and above](https://discuss.elastic.co/t/ability-to-remove-tags-field-from-output-broke-in-5-0-1-and-above/69206)
**Author:** [@ronnocol](https://discuss.elastic.co/u/ronnocol)\ **Replies:** 1\ **Last updated:** [December 16, 2016, 4:30pm UTC](https://discuss.elastic.co/t/ability-to-remove-tags-field-from-output-broke-in-5-0-1-and-above/69206 "2016-12-16T16:30:03Z")
I'm trying to output to a very picky HTTP API, and the existence of a field in the JSON that is posted causes it confusion. I originally did the integration work with 5.0.0 (latest at the time) but then when we went to d… --- ## [How to add new line in between string field](https://discuss.elastic.co/t/how-to-add-new-line-in-between-string-field/40257)
**Author:** [@srujani](https://discuss.elastic.co/u/srujani)\ **Replies:** 1\ **Last updated:** [December 16, 2016, 1:48pm UTC](https://discuss.elastic.co/t/how-to-add-new-line-in-between-string-field/40257 "2016-12-16T13:48:49Z")
Hi, Added the below configuration in logstash mutate { add\_field =\> \["temp", "Adapter Name: %{adaptername} Records Imported: %{records} Records Failed: %{\[failedrecords\]\[1\]} "\] i would like to display th… --- ## [Logstash having exception while starting](https://discuss.elastic.co/t/logstash-having-exception-while-starting/69173)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 6\ **Last updated:** [December 16, 2016, 12:33pm UTC](https://discuss.elastic.co/t/logstash-having-exception-while-starting/69173 "2016-12-16T12:33:29Z")
I am running in V5.0.1 It looks like my logstash having errors and exceptions when booting. Below the link to log: https://gist.github.com/ssasporta/4da3e467a5d95381f14476b204a923af Thanks Sharon. --- ## [Grok "?" operator](https://discuss.elastic.co/t/grok-operator/68653)
**Author:** [@sancroth](https://discuss.elastic.co/u/sancroth)\ **Replies:** 7\ **Last updated:** [December 16, 2016, 7:35am UTC](https://discuss.elastic.co/t/grok-operator/68653 "2016-12-16T07:35:18Z")
Hello. I am using a grok filter on certain logs that give me a problem. A small part of them come out with one less field than the others. example given : input log 1 num=10 event=attack source=mach1 input log 2 … --- ## [Logstash skipping/missing event in case of file rotation](https://discuss.elastic.co/t/logstash-skipping-missing-event-in-case-of-file-rotation/69202)
**Author:** [@Ankit\_Jain](https://discuss.elastic.co/u/Ankit_Jain)\ **Replies:** 2\ **Last updated:** [December 16, 2016, 5:51am UTC](https://discuss.elastic.co/t/logstash-skipping-missing-event-in-case-of-file-rotation/69202 "2016-12-16T05:51:37Z")
Hi, We are reading log files which are rotating after every 11 MB. All the logs are logging in x.log file and the file is rotate to X.log.date (renamed) after every 11 MB size. According to our observation the logstas… --- ## [Concat the XML array in logstash](https://discuss.elastic.co/t/concat-the-xml-array-in-logstash/69241)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 0\ **Last updated:** [December 16, 2016, 6:05am UTC](https://discuss.elastic.co/t/concat-the-xml-array-in-logstash/69241 "2016-12-16T06:05:52Z")
HI Team, I'm just working on new type of XML file. I have some array information in that file i want to gather all those data.. This array fields are repeating with same tags due to this while processing the log the i'm … --- ## [Spilt an Event Into 2](https://discuss.elastic.co/t/spilt-an-event-into-2/69219)
**Author:** [@Nastooh](https://discuss.elastic.co/u/Nastooh)\ **Replies:** 2\ **Last updated:** [December 15, 2016, 11:47pm UTC](https://discuss.elastic.co/t/spilt-an-event-into-2/69219 "2016-12-15T23:47:44Z")
Hi I've been looking at similar QA on similar topic, and couldn't find an applicable answer. I think I have a simple case, i.e., I get records of form: who="Mr. boo", A1=v1,A2=v2 and what I would like to get are 2 record… --- ## [Link log4j to a logstash docker image](https://discuss.elastic.co/t/link-log4j-to-a-logstash-docker-image/69207)
**Author:** [@ImInLoveWithCacao](https://discuss.elastic.co/u/ImInLoveWithCacao)\ **Replies:** 0\ **Last updated:** [December 15, 2016, 6:33pm UTC](https://discuss.elastic.co/t/link-log4j-to-a-logstash-docker-image/69207 "2016-12-15T18:33:46Z")
Hi there, I'm currently trying to send my log4j logs to a docker image of logstash that is running on a vm. The logstash docker image works fine by itself (when I set stdin{} or file{} as input) and log4j also works fi… --- ## [Possible to grok an Elasticsearch document fields?](https://discuss.elastic.co/t/possible-to-grok-an-elasticsearch-document-fields/69039)
**Author:** [@Dom-nik](https://discuss.elastic.co/u/Dom-nik)\ **Replies:** 3\ **Last updated:** [December 15, 2016, 5:23pm UTC](https://discuss.elastic.co/t/possible-to-grok-an-elasticsearch-document-fields/69039 "2016-12-15T17:23:01Z")
Hello! I'm trying to load some data from an ES index (using Elasticsearch input plugin), parse some of its fields with grok filter and load them back into another ES index. I can't seem to find a way to access a particul… --- ## [JSON date conversion](https://discuss.elastic.co/t/json-date-conversion/68967)
**Author:** [@shambar](https://discuss.elastic.co/u/shambar)\ **Replies:** 5\ **Last updated:** [December 15, 2016, 4:05pm UTC](https://discuss.elastic.co/t/json-date-conversion/68967 "2016-12-15T16:05:38Z")
Is there an easy way for logstash to convert a date form json date format to a readable format? and put it back in the same variable? the date comes in like {"srcdate":"\\/Date(1164258000000)\\/"} I would like to … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1062) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1064)