# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=953 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 954 --- ## [Elapsed filter inquiry](https://discuss.elastic.co/t/elapsed-filter-inquiry/90118)
**Author:** [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\ **Replies:** 4\ **Last updated:** [July 3, 2017, 9:34am UTC](https://discuss.elastic.co/t/elapsed-filter-inquiry/90118 "2017-07-03T09:34:59Z")
Hello, I created an elapsed filter, but I can't find the time elapsed in the stdout neither elasticsearch/kibana, I only find those lines in stdout: logstash\_1 | \[2017-06-20T15:12:48,787\]\[INFO \]\[logstash.filters.… --- ## [How to convert long number format datetime to datetime?](https://discuss.elastic.co/t/how-to-convert-long-number-format-datetime-to-datetime/91609)
**Author:** [@sonnguyenit](https://discuss.elastic.co/u/sonnguyenit)\ **Replies:** 3\ **Last updated:** [July 3, 2017, 9:26am UTC](https://discuss.elastic.co/t/how-to-convert-long-number-format-datetime-to-datetime/91609 "2017-07-03T09:26:39Z")
I have a field that is datetime in long value: "action\_date":636346116012468640 So i want to use date filter to convert it to Unix date time: date { match =\> \[ "action\_date", "what is date format?" \] } … --- ## [Date Parsing the wrong Day](https://discuss.elastic.co/t/date-parsing-the-wrong-day/91434)
**Author:** [@Lukas\_Tilch](https://discuss.elastic.co/u/Lukas_Tilch)\ **Replies:** 2\ **Last updated:** [July 3, 2017, 9:00am UTC](https://discuss.elastic.co/t/date-parsing-the-wrong-day/91434 "2017-07-03T09:00:12Z")
Hello, I have a rather unexpected problem with Logstashs Date filter. I want to parse the Date 2017.05.30 with the YYYY.MM.dd I expected it to be parsed as 2017-05-30T00:00:00.000Z Instead it looks like this 2017-05-29… --- ## [IMAP input CSV filter](https://discuss.elastic.co/t/imap-input-csv-filter/91468)
**Author:** [@gurpal](https://discuss.elastic.co/u/gurpal)\ **Replies:** 2\ **Last updated:** [July 3, 2017, 7:53am UTC](https://discuss.elastic.co/t/imap-input-csv-filter/91468 "2017-07-03T07:53:38Z")
Hi there, I have setup IMAP as an input - this works fine as I get the email. The e-mail content is plain text and a simple CSV file: eg Thu Feb 09 21:01:04 2017,28.6,87.2,Fridge 10283739,2.7895,-54 Thu Feb 09 21:1… --- ## [Matching log Using grok pattern](https://discuss.elastic.co/t/matching-log-using-grok-pattern/91578)
**Author:** [@0314e7ff87c297426a9e](https://discuss.elastic.co/u/0314e7ff87c297426a9e)\ **Replies:** 1\ **Last updated:** [July 3, 2017, 7:45am UTC](https://discuss.elastic.co/t/matching-log-using-grok-pattern/91578 "2017-07-03T07:45:38Z")
i can't match log - field using grok patten in logstash Log 2015-06-27 01:32:00, 002 DEBUG \[pool-2-thread-1 \[container11-13\]\] db.QueryInterceptor (QueryInterceptor.java:57) - @@@@@@@@@ query @@@@@@@@@@ SELECT … --- ## [Logstash file input read order](https://discuss.elastic.co/t/logstash-file-input-read-order/90246)
**Author:** [@giannisapi](https://discuss.elastic.co/u/giannisapi)\ **Replies:** 5\ **Last updated:** [July 3, 2017, 7:43am UTC](https://discuss.elastic.co/t/logstash-file-input-read-order/90246 "2017-07-03T07:43:20Z")
Hi to All, I have one configuration file with multiple file inputs. Since deploy we have parsed hundreds of files, I now want to re parse everything but I want to re parse them ordered by creation time. Could you t… --- ## [TCP output reconnect interval setting](https://discuss.elastic.co/t/tcp-output-reconnect-interval-setting/90885)
**Author:** [@s802645](https://discuss.elastic.co/u/s802645)\ **Replies:** 9\ **Last updated:** [July 3, 2017, 7:43am UTC](https://discuss.elastic.co/t/tcp-output-reconnect-interval-setting/90885 "2017-07-03T07:43:06Z")
The reconnect interval setting for TCP output plugin is by default 10. What would happen if I set it to 0. Does it mean logstash would not attempt to reconnect to the destination at all? thanks. --- ## [Removing duplications in output block](https://discuss.elastic.co/t/removing-duplications-in-output-block/91549)
**Author:** [@BentCoder](https://discuss.elastic.co/u/BentCoder)\ **Replies:** 1\ **Last updated:** [July 3, 2017, 6:18am UTC](https://discuss.elastic.co/t/removing-duplications-in-output-block/91549 "2017-07-03T06:18:30Z")
Hi, Any way of slimming output down? Duplication looks ugly. For example host, sniffing and manage\_template could be made common/shared. Thanks output { if \[type\] == "apache-access" { elasticsearch { … --- ## [Grok Filter help needed with Suricata Logs](https://discuss.elastic.co/t/grok-filter-help-needed-with-suricata-logs/90727)
**Author:** [@Malik\_Mazhar](https://discuss.elastic.co/u/Malik_Mazhar)\ **Replies:** 4\ **Last updated:** [July 3, 2017, 5:46am UTC](https://discuss.elastic.co/t/grok-filter-help-needed-with-suricata-logs/90727 "2017-07-03T05:46:39Z")
Hi Guys, i am trying to sort my Suricata logs on pfsense, got it working but one issue i cant seperate Priority from the number and proto from the {}.. below is the sample log need your help. \[1:2210045:2\] SURICATA ST… --- ## [Unable to convert field from string to integer](https://discuss.elastic.co/t/unable-to-convert-field-from-string-to-integer/38915)
**Author:** [@echerno21](https://discuss.elastic.co/u/echerno21)\ **Replies:** 6\ **Last updated:** [July 2, 2017, 11:56pm UTC](https://discuss.elastic.co/t/unable-to-convert-field-from-string-to-integer/38915 "2017-07-02T23:56:52Z")
I am unable to convert a field, that I am generating using 'kv' plugin, from string to integer. Here is the snippet of the config: filter { kv { allow\_duplicate\_values =\> false value\_split =\> "=" } mutate { conver… --- ## [Elapsed filter is not working](https://discuss.elastic.co/t/elapsed-filter-is-not-working/91530)
**Author:** [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\ **Replies:** 3\ **Last updated:** [July 2, 2017, 9:08pm UTC](https://discuss.elastic.co/t/elapsed-filter-is-not-working/91530 "2017-07-02T21:08:09Z")
Hello, I am trying to use Elapsed filter to get the duration between two timestamps, here's a snap of my logfile: 2017-01-01 07:53:44 \[utils.py\] WARNING: enable\_proxy must have atleast one http 2017-01-01 07:53:45 \[pro… --- ## [Csv parsing error, like in an out of buffer error \[solved\]](https://discuss.elastic.co/t/csv-parsing-error-like-in-an-out-of-buffer-error-solved/91566)
**Author:** [@dfermiumlabs](https://discuss.elastic.co/u/dfermiumlabs)\ **Replies:** 2\ **Last updated:** [July 2, 2017, 5:43pm UTC](https://discuss.elastic.co/t/csv-parsing-error-like-in-an-out-of-buffer-error-solved/91566 "2017-07-02T17:43:04Z")
Hy guys, i'm having a really hard time diagnosing this. I'm getting tons of tons of parsing errors where it appears logstash is sending to ES completely out of bound fields. For example, it's supposed to send a number … --- ## [Data from Mysql JSON column to Elasticsearch](https://discuss.elastic.co/t/data-from-mysql-json-column-to-elasticsearch/91559)
**Author:** [@Eduard\_Kubanda](https://discuss.elastic.co/u/Eduard_Kubanda)\ **Replies:** 1\ **Last updated:** [July 2, 2017, 5:28pm UTC](https://discuss.elastic.co/t/data-from-mysql-json-column-to-elasticsearch/91559 "2017-07-02T17:28:47Z")
Hi there, I checked Logstash reference and some tutorials about Mysql -\> Logstash -\> Elasticsearch data migration scenario, but I still have some questions to answer. (Mysql 5.7, Elasticsearch 5.4) I have table "pre\_buil… --- ## [Dateparse error out of the blue](https://discuss.elastic.co/t/dateparse-error-out-of-the-blue/91551)
**Author:** [@kottapar](https://discuss.elastic.co/u/kottapar)\ **Replies:** 1\ **Last updated:** [July 2, 2017, 10:50am UTC](https://discuss.elastic.co/t/dateparse-error-out-of-the-blue/91551 "2017-07-02T10:50:12Z")
Hey guys, The logstash filter for the ESX messages was working fine. However out of the blue it started throwing \_dateparse errors. { "msg" =\> "bound to 16.54.5.10 -- renewal in 255 seconds.", … --- ## [Redis or rabbbitmq](https://discuss.elastic.co/t/redis-or-rabbbitmq/91543)
**Author:** [@prash](https://discuss.elastic.co/u/prash)\ **Replies:** 1\ **Last updated:** [July 2, 2017, 8:59am UTC](https://discuss.elastic.co/t/redis-or-rabbbitmq/91543 "2017-07-02T08:59:30Z")
Continuing the discussion from Syslog-ng + Logstash + Splunk: --- ## [Increase logstash Events Received Rate when using kafka as input](https://discuss.elastic.co/t/increase-logstash-events-received-rate-when-using-kafka-as-input/91507)
**Author:** [@hossein\_ey](https://discuss.elastic.co/u/hossein_ey)\ **Replies:** 2\ **Last updated:** [July 1, 2017, 8:58am UTC](https://discuss.elastic.co/t/increase-logstash-events-received-rate-when-using-kafka-as-input/91507 "2017-07-01T08:58:21Z")
I develop a system with Apache Kafka and Logstash And ElasticSearch. data Produced into Kafka and logstash indexes data into ElasticSearch. In Kafka side, I have good throughput and Produce 30K message per second, this… --- ## [LogStash , Kafka , multiple Topics , rebalancing one topic](https://discuss.elastic.co/t/logstash-kafka-multiple-topics-rebalancing-one-topic/91505)
**Author:** [@Kim-Kruse-Hansen](https://discuss.elastic.co/u/Kim-Kruse-Hansen)\ **Replies:** 0\ **Last updated:** [July 1, 2017, 7:57am UTC](https://discuss.elastic.co/t/logstash-kafka-multiple-topics-rebalancing-one-topic/91505 "2017-07-01T07:57:15Z")
Hi I have a 3 ELK cluster 5.4.1 with 3 nodes , 12 CPU , 32 GB RAM , windows 2012 R2 , VM on VMWare. They pull data from Kafka. The sources are windows log events , dns logs and syslog from network devices. windows logs… --- ## [Syntactic error on logstash config](https://discuss.elastic.co/t/syntactic-error-on-logstash-config/91466)
**Author:** [@Zarkopafilis](https://discuss.elastic.co/u/Zarkopafilis)\ **Replies:** 3\ **Last updated:** [July 1, 2017, 6:20am UTC](https://discuss.elastic.co/t/syntactic-error-on-logstash-config/91466 "2017-07-01T06:20:46Z")
\[2017-06-30T19:06:06,714\]\[DEBUG\]\[logstash.agent \] Reading config file {:config\_file=\>"/etc/logstash/conf.d/logstash.conf"} \[2017-06-30T19:06:06,902\]\[ERROR\]\[logstash.agent \] Cannot create pipeline {:re… --- ## [Netflow Logstash V9 not working not getting any logs](https://discuss.elastic.co/t/netflow-logstash-v9-not-working-not-getting-any-logs/91228)
**Author:** [@philippkahr](https://discuss.elastic.co/u/philippkahr)\ **Replies:** 10\ **Last updated:** [June 30, 2017, 7:20pm UTC](https://discuss.elastic.co/t/netflow-logstash-v9-not-working-not-getting-any-logs/91228 "2017-06-30T19:20:58Z")
Hallo Guys, i face a really weird issue. I cannot get any netflow logs into logstash. They even dont show up on stdout. i can see with tcpdump dst port 1535 that there are logs incoming. With tcpdump dst port 1535 -A i… --- ## [Newbie question syslog fluentd \> kibana (via elastic)](https://discuss.elastic.co/t/newbie-question-syslog-fluentd-kibana-via-elastic/91436)
**Author:** [@zippytiff](https://discuss.elastic.co/u/zippytiff)\ **Replies:** 2\ **Last updated:** [June 30, 2017, 3:46pm UTC](https://discuss.elastic.co/t/newbie-question-syslog-fluentd-kibana-via-elastic/91436 "2017-06-30T15:46:44Z")
Hi, Thanks for reading, So I'm very new to this, a few work hours in. I'm looking to be able to visualise my syslogs from a growing hardware security platform. I have got the basics working... Hardware \> Fluent… --- ## [Logstash error: Cannot create pipeline](https://discuss.elastic.co/t/logstash-error-cannot-create-pipeline/91439)
**Author:** [@CDR](https://discuss.elastic.co/u/CDR)\ **Replies:** 1\ **Last updated:** [June 30, 2017, 1:59pm UTC](https://discuss.elastic.co/t/logstash-error-cannot-create-pipeline/91439 "2017-06-30T13:59:16Z")
I am running logstash using docker. When I ran my docker-compose up I got this error: 13:31:16.723 \[LogStash::Runner\] ERROR logstash.agent - Cannot create pipeline {:reason=\>"Expected one of #, input, filter, output at… --- ## [Two copy of one log](https://discuss.elastic.co/t/two-copy-of-one-log/91139)
**Author:** [@erion](https://discuss.elastic.co/u/erion)\ **Replies:** 12\ **Last updated:** [June 30, 2017, 1:57pm UTC](https://discuss.elastic.co/t/two-copy-of-one-log/91139 "2017-06-30T13:57:13Z")
Hi, when i upload a string of apache log in kibana i see one with real timestamp and the same string with timestamp of uploaded . I use the mapping founded on github here https://github.com/elastic/examples/tree/master/… --- ## [Issues with GREEDYDATA](https://discuss.elastic.co/t/issues-with-greedydata/91431)
**Author:** [@lucian](https://discuss.elastic.co/u/lucian)\ **Replies:** 0\ **Last updated:** [June 30, 2017, 1:02pm UTC](https://discuss.elastic.co/t/issues-with-greedydata/91431 "2017-06-30T13:02:36Z")
Hi, I am new in logstash. I try to translate a logline like this: 2017-06-30 14:49:21 aaa\_BBB12345 BBB1111111111111111111111111111 ERROR Received file not found: /path/to/somewhere/1.txt I use the folowing grok: (%{TIME… --- ## [Logstash filter plugin development flow with SQLite](https://discuss.elastic.co/t/logstash-filter-plugin-development-flow-with-sqlite/89940)
**Author:** [@nacastroc](https://discuss.elastic.co/u/nacastroc)\ **Replies:** 3\ **Last updated:** [June 30, 2017, 12:41pm UTC](https://discuss.elastic.co/t/logstash-filter-plugin-development-flow-with-sqlite/89940 "2017-06-30T12:41:39Z")
I'm a newbie at Logstash plugin development and, as both a work task and a training exercise, I'm developing a filter plugin that involves querying an SQLite database on each event. As of this moment, I understand the fo… --- ## [Trouble with logstash log4j2 rotation](https://discuss.elastic.co/t/trouble-with-logstash-log4j2-rotation/91412)
**Author:** [@jonathanaxel](https://discuss.elastic.co/u/jonathanaxel)\ **Replies:** 1\ **Last updated:** [June 30, 2017, 11:24am UTC](https://discuss.elastic.co/t/trouble-with-logstash-log4j2-rotation/91412 "2017-06-30T11:24:23Z")
Hi all, I'm trying to enable the logstash log rotation based on size, because of the big stream of data the logs are growing in size fast. This is my log4j2.properties: status = error name = LogstashPropertiesCo… --- ## [Output of geoip filter changed in logstash 5.4.2 (no more GeoJSON)](https://discuss.elastic.co/t/output-of-geoip-filter-changed-in-logstash-5-4-2-no-more-geojson/91186)
**Author:** [@matejzero](https://discuss.elastic.co/u/matejzero)\ **Replies:** 3\ **Last updated:** [June 30, 2017, 10:01am UTC](https://discuss.elastic.co/t/output-of-geoip-filter-changed-in-logstash-5-4-2-no-more-geojson/91186 "2017-06-30T10:01:07Z")
Hello, I'm testing logstash 5.4.2 in my staging env and I've noticed that output of geoip plugins has changes. Instead of GeoJSON output, like I had in 5.4.1 "location": \[ -122.3042, 47.913 \], I… --- ## [Logstash repeatedly insert same data from corrupted S3 bucket file](https://discuss.elastic.co/t/logstash-repeatedly-insert-same-data-from-corrupted-s3-bucket-file/91369)
**Author:** [@Sunil\_M](https://discuss.elastic.co/u/Sunil_M)\ **Replies:** 1\ **Last updated:** [June 30, 2017, 9:32am UTC](https://discuss.elastic.co/t/logstash-repeatedly-insert-same-data-from-corrupted-s3-bucket-file/91369 "2017-06-30T09:32:12Z")
Hi, I am using logstash to read S3 bucket file and insert to Elasticsearch. I am facing 2 issues right now. If the S3 file is empty, the logstash does not delete and move forward rather it fires error. If the file … --- ## [Error registering plugin](https://discuss.elastic.co/t/error-registering-plugin/89901)
**Author:** [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\ **Replies:** 3\ **Last updated:** [June 30, 2017, 9:31am UTC](https://discuss.elastic.co/t/error-registering-plugin/89901 "2017-06-30T09:31:14Z")
Hey guys ! I got one thing strange in my ELK conf. When I launch my logstash commpand, the pipeline seems to be created with no issuses (i can see the overview in the monitoring menu of Kibana), but I got this erro… --- ## [Logstash Core Plugin Firewall Patterns are outdated](https://discuss.elastic.co/t/logstash-core-plugin-firewall-patterns-are-outdated/91326)
**Author:** [@DanMa](https://discuss.elastic.co/u/DanMa)\ **Replies:** 1\ **Last updated:** [June 30, 2017, 9:24am UTC](https://discuss.elastic.co/t/logstash-core-plugin-firewall-patterns-are-outdated/91326 "2017-06-30T09:24:15Z")
Hey, iam trying to built up a Syslog Solution for my ASA Firewall. I use udp as Input.. I use this grok filter (and i get everytime a \_grokparsefailure): %{CISCO\_TAGGED\_SYSLOG}%{GREEDYDATA:cisco\_message} After my… --- ## [The best practice to nest a hash?](https://discuss.elastic.co/t/the-best-practice-to-nest-a-hash/91373)
**Author:** [@lucianspec](https://discuss.elastic.co/u/lucianspec)\ **Replies:** 4\ **Last updated:** [June 30, 2017, 9:20am UTC](https://discuss.elastic.co/t/the-best-practice-to-nest-a-hash/91373 "2017-06-30T09:20:01Z")
my event data like this... "thread" =\> "Thread-74-\_\_system-executor\[-1 -1\]", "message" =\> "2017-06-19 16:46:12.750 o.a.s.d.executor Thread-74-\_\_system-executor\[-1 -1\] \[INFO\] Prepared bolt \_\_system:(-1)", "… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=952) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=954)