# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=388 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 389 --- ## [Kafka input plugin compressed data](https://discuss.elastic.co/t/kafka-input-plugin-compressed-data/224114)
**Author:** [@Dhruvil\_Doshi](https://discuss.elastic.co/u/Dhruvil_Doshi)\ **Replies:** 0\ **Last updated:** [March 18, 2020, 1:19pm UTC](https://discuss.elastic.co/t/kafka-input-plugin-compressed-data/224114 "2020-03-18T13:19:33Z")
In logstash, I am taking input from kafka. The messages in kafka are compressed. So I am getting compressed data in my elasticsearch index. Is there an way to decompress data in logstash. Data is compressed using gzip T… --- ## [Convert Number (elapsed\_time) into Time (HH:MM:SS)](https://discuss.elastic.co/t/convert-number-elapsed-time-into-time-hhss/218551)
**Author:** [@Thibaut\_M](https://discuss.elastic.co/u/Thibaut_M)\ **Replies:** 4\ **Last updated:** [March 18, 2020, 12:58pm UTC](https://discuss.elastic.co/t/convert-number-elapsed-time-into-time-hhss/218551 "2020-03-18T12:58:34Z")
Hello, Let me expose what i'm trying to achieve here. I calculate an elapsed\_time (with the elapsed plugin) between two documents. elapsed\_time is in second and i want it like this : HH:MM:SS so as an example : 80 sec =… --- ## [Parsing logs from Fortinet WAF](https://discuss.elastic.co/t/parsing-logs-from-fortinet-waf/222794)
**Author:** [@david-vazquez](https://discuss.elastic.co/u/david-vazquez)\ **Replies:** 16\ **Last updated:** [March 18, 2020, 12:25pm UTC](https://discuss.elastic.co/t/parsing-logs-from-fortinet-waf/222794 "2020-03-18T12:25:46Z")
Hello all, Im new using ELK, so I have some doubts in some stuff I´m trying to do. I configured ELK Stack and im trying to ingest some logs to Elasticsearch. One of them are logs from Fortinet Web Application Firewall. … --- ## [Best practice logstash/filebeat](https://discuss.elastic.co/t/best-practice-logstash-filebeat/224078)
**Author:** [@ledoc](https://discuss.elastic.co/u/ledoc)\ **Replies:** 0\ **Last updated:** [March 18, 2020, 10:49am UTC](https://discuss.elastic.co/t/best-practice-logstash-filebeat/224078 "2020-03-18T10:49:40Z")
Hello there, I have question on best practice of logstash, and workspace in kibana. On my ELK test environment, I have filebeat/nxlog/gelf \> logstash \> elasticsearch (standard configuration). First, I have a lot of … --- ## [Separate indexes for grok filters](https://discuss.elastic.co/t/separate-indexes-for-grok-filters/224034)
**Author:** [@nikhilesh](https://discuss.elastic.co/u/nikhilesh)\ **Replies:** 3\ **Last updated:** [March 18, 2020, 8:35am UTC](https://discuss.elastic.co/t/separate-indexes-for-grok-filters/224034 "2020-03-18T08:35:00Z")
I am wondering how to create separated indexes for different logs fetched into logstash (which were later passed onto elasticsearch), so that in kibana, In my case, I have a few client servers (each of which is installe… --- ## [Logstash docker container failed to start found this error in /var/log/messages mounted filesystem with ordered data mode. Opts: (null)](https://discuss.elastic.co/t/logstash-docker-container-failed-to-start-found-this-error-in-var-log-messages-mounted-filesystem-with-ordered-data-mode-opts-null/223931)
**Author:** [@Krish\_7](https://discuss.elastic.co/u/Krish_7)\ **Replies:** 0\ **Last updated:** [March 17, 2020, 1:26pm UTC](https://discuss.elastic.co/t/logstash-docker-container-failed-to-start-found-this-error-in-var-log-messages-mounted-filesystem-with-ordered-data-mode-opts-null/223931 "2020-03-17T13:26:23Z")
Hi Team, I am using Docker version 17.05.0-ce, build 89658be I have logstash container hosted which is scheduled to be stop and start every day. Which is getting failed to start in the first attempt i have checked /va… --- ## [Message queue for ELK Stack on Windows](https://discuss.elastic.co/t/message-queue-for-elk-stack-on-windows/224025)
**Author:** [@nityaraj06](https://discuss.elastic.co/u/nityaraj06)\ **Replies:** 0\ **Last updated:** [March 18, 2020, 4:51am UTC](https://discuss.elastic.co/t/message-queue-for-elk-stack-on-windows/224025 "2020-03-18T04:51:33Z")
Hi, I have been exploring about adding a message queue to my ELK stack on Windows server 2016 Redis seems to be a good option to use but it seems like it is not compatible with Windows. Could you please suggest a messa… --- ## [Split Json Array Objects](https://discuss.elastic.co/t/split-json-array-objects/223088)
**Author:** [@Atul\_Gunjal](https://discuss.elastic.co/u/Atul_Gunjal)\ **Replies:** 8\ **Last updated:** [March 18, 2020, 4:03am UTC](https://discuss.elastic.co/t/split-json-array-objects/223088 "2020-03-18T04:03:21Z")
Hi, I want to split my json array into multiple events in logstash. I tried using split field but it doesn't seem to work that way for objects in array. So any reference which plugin should I use in logstash to process … --- ## [Logstash xpack.monitoring error for endpoint \_bulk with TLS/SSL](https://discuss.elastic.co/t/logstash-xpack-monitoring-error-for-endpoint-bulk-with-tls-ssl/223600)
**Author:** [@danksim](https://discuss.elastic.co/u/danksim)\ **Replies:** 2\ **Last updated:** [March 17, 2020, 8:37pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-error-for-endpoint-bulk-with-tls-ssl/223600 "2020-03-17T20:37:44Z")
Using images from elastic/helm-charts elasticsearch version: 6.8.2 logstash version: 6.7.0 Ever since enabling TLS for all ELK components, I am seeing the following error in logstash pod logs: \[2020-03-14T04:05:50,1… --- ## [Issue with output elasticsearch running on ECK "HostUnreachableError"](https://discuss.elastic.co/t/issue-with-output-elasticsearch-running-on-eck-hostunreachableerror/223109)
**Author:** [@Xavier\_Krantz](https://discuss.elastic.co/u/Xavier_Krantz)\ **Replies:** 4\ **Last updated:** [March 17, 2020, 4:41pm UTC](https://discuss.elastic.co/t/issue-with-output-elasticsearch-running-on-eck-hostunreachableerror/223109 "2020-03-17T16:41:08Z")
Hi there ! So first of all, thank your for reading my post. I have already browsed the existing topics on discuss about that kind of issue and that is what I have tried without any success: Setup ssl\_certificate\_veri… --- ## [Logstash: per-pipeline logfiles](https://discuss.elastic.co/t/logstash-per-pipeline-logfiles/195379)
**Author:** [@barnetta](https://discuss.elastic.co/u/barnetta)\ **Replies:** 3\ **Last updated:** [March 17, 2020, 3:39pm UTC](https://discuss.elastic.co/t/logstash-per-pipeline-logfiles/195379 "2020-03-17T15:39:40Z")
I have a Logstash development environment with multiple pipelines. From time to time, I accidentally introduce an error into the config. In such a case, I would find it very useful to have a per-pipeline logfile, in addi… --- ## [Parsing non-JSON key - lists-of-values pairs](https://discuss.elastic.co/t/parsing-non-json-key-lists-of-values-pairs/223956)
**Author:** [@Antonio\_Bevilacqua](https://discuss.elastic.co/u/Antonio_Bevilacqua)\ **Replies:** 0\ **Last updated:** [March 17, 2020, 2:57pm UTC](https://discuss.elastic.co/t/parsing-non-json-key-lists-of-values-pairs/223956 "2020-03-17T14:57:24Z")
I am trying to parsing with a logstash filter the following pattern: \[ \[k1, \[ v11, v12, v13 \] \], \[ k2, \[ v21 \] \], \[k3, \[ v31, v32 \] \] \] This is a list of key value pairs with varying length, where the values are lists … --- ## [Basic ruby field adding not working](https://discuss.elastic.co/t/basic-ruby-field-adding-not-working/223853)
**Author:** [@imrago](https://discuss.elastic.co/u/imrago)\ **Replies:** 2\ **Last updated:** [March 17, 2020, 2:22pm UTC](https://discuss.elastic.co/t/basic-ruby-field-adding-not-working/223853 "2020-03-17T14:22:07Z")
Hi, attempted to add a new field with ruby and not even a simple adding is working, what could be wrong? input { beats { port =\> 5044 } } filter { ruby { code =\> "event\['testfield'\] = 'test'" } } output { e… --- ## [Rename fieldnames based on an import configuration table](https://discuss.elastic.co/t/rename-fieldnames-based-on-an-import-configuration-table/223938)
**Author:** [@Pascal\_Rheinert](https://discuss.elastic.co/u/Pascal_Rheinert)\ **Replies:** 0\ **Last updated:** [March 17, 2020, 1:40pm UTC](https://discuss.elastic.co/t/rename-fieldnames-based-on-an-import-configuration-table/223938 "2020-03-17T13:40:20Z")
I have cassandra database with two tables "products" and "fields". The product table consists of the following fields: "ID", "1:Value", "2:Value", .. "99:Value" Example: "ID", "1:Value", "2:Value", "3:Value" 1, Test,… --- ## [Prevent Logstash Service logging to /var/log/syslog](https://discuss.elastic.co/t/prevent-logstash-service-logging-to-var-log-syslog/223875)
**Author:** [@Amish](https://discuss.elastic.co/u/Amish)\ **Replies:** 1\ **Last updated:** [March 17, 2020, 12:47pm UTC](https://discuss.elastic.co/t/prevent-logstash-service-logging-to-var-log-syslog/223875 "2020-03-17T12:47:33Z")
Hi, I am trying to work out how and where I can stop the logging of Logstash to the syslog on Ubuntu 16.04. Instead I would like to have all of the logs for logs for Logstash in a separate file. I should mention that I… --- ## [Emailing through logstash not working](https://discuss.elastic.co/t/emailing-through-logstash-not-working/223690)
**Author:** [@Sagar\_Mandal](https://discuss.elastic.co/u/Sagar_Mandal)\ **Replies:** 3\ **Last updated:** [March 17, 2020, 12:41pm UTC](https://discuss.elastic.co/t/emailing-through-logstash-not-working/223690 "2020-03-17T12:41:26Z")
Hi Team, I wrote this small logstash config. in which i just want to pick up some data and when the data comes it gets filter, and in the output section when the data comes it send an email. input{ file { type … --- ## [How can i filter this pattern?](https://discuss.elastic.co/t/how-can-i-filter-this-pattern/223885)
**Author:** [@Ramon\_Mateo](https://discuss.elastic.co/u/Ramon_Mateo)\ **Replies:** 8\ **Last updated:** [March 17, 2020, 11:42am UTC](https://discuss.elastic.co/t/how-can-i-filter-this-pattern/223885 "2020-03-17T11:42:26Z")
Hi I try to filter this \> \\"tenantID\\": 80334, \\"Size\\": 11.42, \\"FirstDate\\": \\"15/03/2020 00:09:47\\", \\"LastDate\\": \\"15/03/2020 23:59:17\\" i need this: tenantID: 80334 Size: 11.42 FirstDate: 15/03/2020 00:09:47 … --- ## [Logstash auto-transforming data, rendering it useless](https://discuss.elastic.co/t/logstash-auto-transforming-data-rendering-it-useless/223496)
**Author:** [@Mattness](https://discuss.elastic.co/u/Mattness)\ **Replies:** 2\ **Last updated:** [March 17, 2020, 10:53am UTC](https://discuss.elastic.co/t/logstash-auto-transforming-data-rendering-it-useless/223496 "2020-03-17T10:53:02Z")
Hey there, I am using logstash to write data from my mssql database to elasticsearch. For whatever reason logstash transfrom the db-field "Duration" which contains values in the style of: "00:00:45.4388525" into this:… --- ## [How to include Clinet IP /Hostname in Slowlog via some modification in log4j2.properties](https://discuss.elastic.co/t/how-to-include-clinet-ip-hostname-in-slowlog-via-some-modification-in-log4j2-properties/223905)
**Author:** [@Vinod\_Thorwat](https://discuss.elastic.co/u/Vinod_Thorwat)\ **Replies:** 0\ **Last updated:** [March 17, 2020, 10:45am UTC](https://discuss.elastic.co/t/how-to-include-clinet-ip-hostname-in-slowlog-via-some-modification-in-log4j2-properties/223905 "2020-03-17T10:45:45Z")
\*\*Elasticsearch version\*\* Version: 6.6.2 \*\*Plugins installed\*\* : \[\] opendistro\_alerting opendistro\_performance\_analyzer opendistro\_security opendistro\_sql repository-hdfs \*\*JVM version\*\* 1.8.0\_221 \*\*OS version\*\* Ce… --- ## [Email Output plugin in logstash](https://discuss.elastic.co/t/email-output-plugin-in-logstash/223894)
**Author:** [@Sagar\_Mandal](https://discuss.elastic.co/u/Sagar_Mandal)\ **Replies:** 0\ **Last updated:** [March 17, 2020, 10:04am UTC](https://discuss.elastic.co/t/email-output-plugin-in-logstash/223894 "2020-03-17T10:04:33Z")
Hi Team, I'm getting this error, in Email output plugin in logstash, \[2020-03-17T01:31:33,099\]\[ ERROR \]\[logstash.outputs.email \]\[main\] Something happen while delivering an email {:exception=\>#\ **Author:** [@norup](https://discuss.elastic.co/u/norup)\ **Replies:** 1\ **Last updated:** [March 17, 2020, 9:09am UTC](https://discuss.elastic.co/t/logstash-halted-with-error-connection-reset-by-peer/223713 "2020-03-17T09:09:52Z") My logstash stopped over the weekend for some reason. I currently only have 1 beat. winlogbeat, shipping its logs to it, but if it breaks every time some1 restarts their PC or something, then thats obviously going to be … --- ## [How to parse a csv file which has some field containing seprator (comma) as-values](https://discuss.elastic.co/t/how-to-parse-a-csv-file-which-has-some-field-containing-seprator-comma-as-values/223708)
**Author:** [@Shahid\_Mustafa](https://discuss.elastic.co/u/Shahid_Mustafa)\ **Replies:** 4\ **Last updated:** [March 17, 2020, 8:26am UTC](https://discuss.elastic.co/t/how-to-parse-a-csv-file-which-has-some-field-containing-seprator-comma-as-values/223708 "2020-03-17T08:26:20Z")
sample message - 111,222,333,444,555,val1in6th,val2in6th,777 The sixth column contains a value consisting of commas (val1in6th,val2in6th is a sample value of 6th column). When I use a simple csv filter this message is … --- ## [Logstash not able to connect to ECE 2.4.3 elasticsearch](https://discuss.elastic.co/t/logstash-not-able-to-connect-to-ece-2-4-3-elasticsearch/223821)
**Author:** [@lcarr](https://discuss.elastic.co/u/lcarr)\ **Replies:** 0\ **Last updated:** [March 16, 2020, 9:07pm UTC](https://discuss.elastic.co/t/logstash-not-able-to-connect-to-ece-2-4-3-elasticsearch/223821 "2020-03-16T21:07:52Z")
LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, - I am moving this discussion over to this forum - it was suggested that it was more to do with logstash connecting to ECE 2.4.3 - The error is … --- ## [Split output of fping properly with grok](https://discuss.elastic.co/t/split-output-of-fping-properly-with-grok/223621)
**Author:** [@hitman22](https://discuss.elastic.co/u/hitman22)\ **Replies:** 5\ **Last updated:** [March 16, 2020, 8:02pm UTC](https://discuss.elastic.co/t/split-output-of-fping-properly-with-grok/223621 "2020-03-16T20:02:46Z")
Hello, I am trying to Split a Output from fping with grok, but I am not sure how to do it properly. I have the following output from fping google.de : xmt/rcv/%loss = 16/16/0%, min/avg/max = 23.6/24.3/25.2 I only nee… --- ## [Error indexing using xml plugin](https://discuss.elastic.co/t/error-indexing-using-xml-plugin/223774)
**Author:** [@fionamacd](https://discuss.elastic.co/u/fionamacd)\ **Replies:** 7\ **Last updated:** [March 16, 2020, 7:49pm UTC](https://discuss.elastic.co/t/error-indexing-using-xml-plugin/223774 "2020-03-16T19:49:45Z")
Hi, logstash 6.8.3 ES 6.3 I am getting constant errors in my logstash log about mapping an xml field: Mar 16 13:09:04 ip-10-152-3-140 logstash\[16367\]: \[2020-03-16T13:09:04,614\]\[WARN \]\[logstash.outputs.elasticsearch\] … --- ## [What does event.cancel() do?](https://discuss.elastic.co/t/what-does-event-cancel-do/223482)
**Author:** [@jtboing](https://discuss.elastic.co/u/jtboing)\ **Replies:** 3\ **Last updated:** [March 16, 2020, 4:51pm UTC](https://discuss.elastic.co/t/what-does-event-cancel-do/223482 "2020-03-16T16:51:04Z")
I've recently been using logstash to pass data from the database to elaticsearch. One of the plugins that I'm using is the aggregate filter. In some of the examples, there is event.cancel() that is said to cancel events. … --- ## [Invalid version of beats protocol](https://discuss.elastic.co/t/invalid-version-of-beats-protocol/223769)
**Author:** [@sam6](https://discuss.elastic.co/u/sam6)\ **Replies:** 1\ **Last updated:** [March 16, 2020, 3:17pm UTC](https://discuss.elastic.co/t/invalid-version-of-beats-protocol/223769 "2020-03-16T15:17:52Z")
Hi, TLS is not enabled on the filebeat or logstash config and im seeing the below error when trying to ingest logs from a fllebeats agent: \[2020-03-16T13:12:20,101\]\[INFO \]\[org.logstash.beats.BeatsHandler\]\[main\] \[local:… --- ## [Filter Grook](https://discuss.elastic.co/t/filter-grook/223728)
**Author:** [@Ramon\_Mateo](https://discuss.elastic.co/u/Ramon_Mateo)\ **Replies:** 1\ **Last updated:** [March 16, 2020, 2:42pm UTC](https://discuss.elastic.co/t/filter-grook/223728 "2020-03-16T14:42:50Z")
Hi, i have a little issue, my program send files to logstash with this format: "tenantID": 81883, "Size": 0.1, "FirstDate": "01/23/2020 22:45:32", "LastDate": "123/21/2020 22:45:32" My problem is when logstash receives… --- ## [Logstash update \[7.6.1\] java\_execution](https://discuss.elastic.co/t/logstash-update-7-6-1-java-execution/223779)
**Author:** [@Finu](https://discuss.elastic.co/u/Finu)\ **Replies:** 0\ **Last updated:** [March 16, 2020, 2:40pm UTC](https://discuss.elastic.co/t/logstash-update-7-6-1-java-execution/223779 "2020-03-16T14:40:55Z")
Hello. Got one question to java\_execution flag. We were developing some plugins previously in logstash 7.4.2. Now I try to update it to 7.6.1. I were not able to start it so I started to dig down. Trimming config to … --- ## [@timestamp field is not converting PM time to correct UTC](https://discuss.elastic.co/t/timestamp-field-is-not-converting-pm-time-to-correct-utc/223602)
**Author:** [@Vinutha\_A\_H](https://discuss.elastic.co/u/Vinutha_A_H)\ **Replies:** 3\ **Last updated:** [March 16, 2020, 2:40pm UTC](https://discuss.elastic.co/t/timestamp-field-is-not-converting-pm-time-to-correct-utc/223602 "2020-03-16T14:40:34Z")
Hi, In our logs, we have our date time as below and trying to set the same to timestamp field in logstash. It's giving correct UTC time for AM timings but not converting to right UTC time for PM timings. Logs : the bel… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=387) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=389)