# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=755 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 756 --- ## [Handle SQL JDBC input errors](https://discuss.elastic.co/t/handle-sql-jdbc-input-errors/132633)
**Author:** [@emon1](https://discuss.elastic.co/u/emon1)\ **Replies:** 3\ **Last updated:** [May 22, 2018, 9:14pm UTC](https://discuss.elastic.co/t/handle-sql-jdbc-input-errors/132633 "2018-05-22T21:14:30Z")
Is there a way to handle JDBC input failures so that the main Logstash pipeline does not not stop and restart every time an error is thrown by one input ? --- ## [Error parsing csv invalid file format](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147)
**Author:** [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\ **Replies:** 15\ **Last updated:** [May 22, 2018, 9:09pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147 "2018-05-22T21:09:13Z")
hello all below is the log format and I was using ; as separator and trying to using the following logstash configuration and I am getting parsing error Kindly suggest what can be done thanks in advance csv { separato… --- ## [Grok help splitting username:ip properly](https://discuss.elastic.co/t/grok-help-splitting-username-ip-properly/132859)
**Author:** [@sm00thindian](https://discuss.elastic.co/u/sm00thindian)\ **Replies:** 1\ **Last updated:** [May 22, 2018, 6:45pm UTC](https://discuss.elastic.co/t/grok-help-splitting-username-ip-properly/132859 "2018-05-22T18:45:34Z")
I have log lines like the following: 2018-05-19 17:59:42 - Nagios XI \[16\] a1abc10:192.168.0.10 - User authenticated via Rapid Response 2018-05-20 00:00:03 - Nagios XI \[32\] system:localhost - User submitted a command to … --- ## [How to setup shards and number\_of\_replicas in logstash index](https://discuss.elastic.co/t/how-to-setup-shards-and-number-of-replicas-in-logstash-index/132864)
**Author:** [@William\_Paixao](https://discuss.elastic.co/u/William_Paixao)\ **Replies:** 2\ **Last updated:** [May 22, 2018, 6:13pm UTC](https://discuss.elastic.co/t/how-to-setup-shards-and-number-of-replicas-in-logstash-index/132864 "2018-05-22T18:13:42Z")
Hello guys, There is a away to setup logstash to send shards and number\_of\_replicas to Elasticsearch in your index ? i.e.: input { redis { host =\> 'redis.sample.com' data\_type =\> 'list' key =\> 'logstash' number\_o… --- ## [Pipelines stops grok filter in Logstash](https://discuss.elastic.co/t/pipelines-stops-grok-filter-in-logstash/130676)
**Author:** [@shine](https://discuss.elastic.co/u/shine)\ **Replies:** 6\ **Last updated:** [May 16, 2018, 1:55pm UTC](https://discuss.elastic.co/t/pipelines-stops-grok-filter-in-logstash/130676 "2018-05-16T13:55:58Z")
PROBLEM: The grok filter is skipped when I run two .conf files using pipelines in Logstash. All the data respectively goes into each index but it skips the grok filter. The grok filter works once for the 1st record Logs… --- ## [Syslog Listener Died - TCP/UDP](https://discuss.elastic.co/t/syslog-listener-died-tcp-udp/128885)
**Author:** [@ElevenB2003](https://discuss.elastic.co/u/ElevenB2003)\ **Replies:** 2\ **Last updated:** [May 22, 2018, 4:15pm UTC](https://discuss.elastic.co/t/syslog-listener-died-tcp-udp/128885 "2018-05-22T16:15:18Z")
Hello, I just did a fresh setup of the latest version of the stack (6.2.4) and everything looks good except for my Logstash syslog listeners for TCP and UDP keep dying so I'm unable to ship in any syslog data. Here is … --- ## [Logstash as a service Windows - Without NSSM/Other](https://discuss.elastic.co/t/logstash-as-a-service-windows-without-nssm-other/132849)
**Author:** [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)\ **Replies:** 0\ **Last updated:** [May 22, 2018, 3:59pm UTC](https://discuss.elastic.co/t/logstash-as-a-service-windows-without-nssm-other/132849 "2018-05-22T15:59:42Z")
Hi all, Is it possible to run Logstash as a service in the background on my Windows Server. I understand this is possible using NSSM, and other tools like this. But is there a different method to achieve the same perh… --- ## [Email grok filter](https://discuss.elastic.co/t/email-grok-filter/132669)
**Author:** [@rouchad\_rouchad](https://discuss.elastic.co/u/rouchad_rouchad)\ **Replies:** 6\ **Last updated:** [May 22, 2018, 3:11pm UTC](https://discuss.elastic.co/t/email-grok-filter/132669 "2018-05-22T15:11:33Z")
hello everyone i need i need your help i have a mission which is to make filters for email messages in order to get accurate information like: ip address, subject, date, email, phone, address, etc. using losgtash filter … --- ## [Error when running logstash.bat -f ..\\logstash.conf](https://discuss.elastic.co/t/error-when-running-logstash-bat-f-logstash-conf/131533)
**Author:** [@phil.zac](https://discuss.elastic.co/u/phil.zac)\ **Replies:** 3\ **Last updated:** [May 22, 2018, 2:43pm UTC](https://discuss.elastic.co/t/error-when-running-logstash-bat-f-logstash-conf/131533 "2018-05-22T14:43:20Z")
Hi, This might be a simple error but I cant find the solution for it. When I run - logstash.bat -f ..\\logstash.conf in command prompt, I get the following. '-Xms1g' is not recognized as an internal or external comma… --- ## [What is the logstash gem?](https://discuss.elastic.co/t/what-is-the-logstash-gem/132839)
**Author:** [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Replies:** 0\ **Last updated:** [May 22, 2018, 2:33pm UTC](https://discuss.elastic.co/t/what-is-the-logstash-gem/132839 "2018-05-22T14:33:28Z")
From documentation at https://www.elastic.co/guide/en/logstash/current/\_how\_to\_write\_a\_logstash\_output\_plugin.html source 'https://rubygems.org' gemspec gem "logstash", :github =\> "elastic/logstash", :branch =\> "6.2" B… --- ## [Output plugin for kafka discovery](https://discuss.elastic.co/t/output-plugin-for-kafka-discovery/132458)
**Author:** [@Akash\_Kumar\_Dutta](https://discuss.elastic.co/u/Akash_Kumar_Dutta)\ **Replies:** 2\ **Last updated:** [May 22, 2018, 1:53pm UTC](https://discuss.elastic.co/t/output-plugin-for-kafka-discovery/132458 "2018-05-22T13:53:40Z")
I am fairly new to logstash. I want to send the logs coming to logstash to a kafka cluster (already running as a k8s service). I know I need to configure the output plugin inside a configMap. From what I have seen here,… --- ## [What is flush\_interval in a logstash file plugin output conf file configuration](https://discuss.elastic.co/t/what-is-flush-interval-in-a-logstash-file-plugin-output-conf-file-configuration/132821)
**Author:** [@Avanish\_Kumar](https://discuss.elastic.co/u/Avanish_Kumar)\ **Replies:** 0\ **Last updated:** [May 22, 2018, 12:44pm UTC](https://discuss.elastic.co/t/what-is-flush-interval-in-a-logstash-file-plugin-output-conf-file-configuration/132821 "2018-05-22T12:44:19Z")
what is the purpose of flush\_interval in a logstash file plugin output conf file configuration? There is not much documentation available. Also, if someone can pls explain how it works. --- ## [Logstash - Merge multiple lines into one event](https://discuss.elastic.co/t/logstash-merge-multiple-lines-into-one-event/132573)
**Author:** [@Sunillinus](https://discuss.elastic.co/u/Sunillinus)\ **Replies:** 5\ **Last updated:** [May 22, 2018, 12:26pm UTC](https://discuss.elastic.co/t/logstash-merge-multiple-lines-into-one-event/132573 "2018-05-22T12:26:07Z")
Hi, I want to create one configuration file for logstash such that it should generate an event and upload a whole message whatever is there in one timestamp to other timestamp. Can anyone please help me how to achieve … --- ## [Plugin download not functional (Logstash 6.2.4)](https://discuss.elastic.co/t/plugin-download-not-functional-logstash-6-2-4/132119)
**Author:** [@plexaikm](https://discuss.elastic.co/u/plexaikm)\ **Replies:** 2\ **Last updated:** [May 22, 2018, 10:54am UTC](https://discuss.elastic.co/t/plugin-download-not-functional-logstash-6-2-4/132119 "2018-05-22T10:54:46Z")
Hello I'm trying to install logstash-input-lumberjack (or jms or log4j) plugins on logstash v 6.2.4 and the installation freezes for a minute and drops exception: \[root@es624 logstash\]# bin/logstash-plugin install log… --- ## [Ruby filter - using require caused CPU spike](https://discuss.elastic.co/t/ruby-filter-using-require-caused-cpu-spike/132538)
**Author:** [@blacklobo](https://discuss.elastic.co/u/blacklobo)\ **Replies:** 1\ **Last updated:** [May 22, 2018, 10:44am UTC](https://discuss.elastic.co/t/ruby-filter-using-require-caused-cpu-spike/132538 "2018-05-22T10:44:01Z")
I am curious if others have had this problem and/or if someone understand WHY this occurs. I had the below bit of code in a Logstash instance that was consuming from one Kafka cluster performing some filtering and then … --- ## [How to use multiple index under logstash output in filebeat](https://discuss.elastic.co/t/how-to-use-multiple-index-under-logstash-output-in-filebeat/132472)
**Author:** [@yoga-sara](https://discuss.elastic.co/u/yoga-sara)\ **Replies:** 1\ **Last updated:** [May 22, 2018, 10:43am UTC](https://discuss.elastic.co/t/how-to-use-multiple-index-under-logstash-output-in-filebeat/132472 "2018-05-22T10:43:16Z")
I want to send same data to two index under logstash output. FYR my filebeat.yml (Logstash output) ################## output.logstash: enabled: true hosts: \["IP:Port"\] index: 'common' ################## how to me… --- ## [How to pass encrypted USerName and Password in Logstash](https://discuss.elastic.co/t/how-to-pass-encrypted-username-and-password-in-logstash/132771)
**Author:** [@Sekhar1](https://discuss.elastic.co/u/Sekhar1)\ **Replies:** 3\ **Last updated:** [May 22, 2018, 10:41am UTC](https://discuss.elastic.co/t/how-to-pass-encrypted-username-and-password-in-logstash/132771 "2018-05-22T10:41:37Z")
Elastic Search Version 6.0.0 Logstash version 6.0.0 logstash.conf file input { jdbc { jdbc\_connection\_string =\> "jdbc:postgresql://DbIP:port/DatabaseName" jdbc\_user =\> "dbuser" jdbc\_password =\> "dbpassword" jdbc\_… --- ## [Unable to gunzip file which is created through logstash gzip =\>true , in a file plugin](https://discuss.elastic.co/t/unable-to-gunzip-file-which-is-created-through-logstash-gzip-true-in-a-file-plugin/132784)
**Author:** [@Avanish\_Kumar](https://discuss.elastic.co/u/Avanish_Kumar)\ **Replies:** 0\ **Last updated:** [May 22, 2018, 10:40am UTC](https://discuss.elastic.co/t/unable-to-gunzip-file-which-is-created-through-logstash-gzip-true-in-a-file-plugin/132784 "2018-05-22T10:40:01Z")
We are unable to gunzip compressed file created through logstash. For some file it is successful but for other fails with below error. Request you to address this issue. // \[root@mymachine\]# gunzip application\_2018-05-… --- ## [Expected one of #, { in logstash configuration parsing](https://discuss.elastic.co/t/expected-one-of-in-logstash-configuration-parsing/132479)
**Author:** [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Replies:** 6\ **Last updated:** [May 22, 2018, 10:28am UTC](https://discuss.elastic.co/t/expected-one-of-in-logstash-configuration-parsing/132479 "2018-05-22T10:28:09Z")
Hi, I am receiving the below error of checking logstash parsing: Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one… --- ## [Not able to parse time column in DB table](https://discuss.elastic.co/t/not-able-to-parse-time-column-in-db-table/132759)
**Author:** [@anisen](https://discuss.elastic.co/u/anisen)\ **Replies:** 1\ **Last updated:** [May 22, 2018, 10:20am UTC](https://discuss.elastic.co/t/not-able-to-parse-time-column-in-db-table/132759 "2018-05-22T10:20:00Z")
Hi, I am trying to parse the data from a Oracle DB table with has a column named ''executiontime", which I want to use to plot the graphs. Value of the column is like : 21-MAY-18 04.24.29.000000000 AM -05:00 using the… --- ## [Do something when Logstash has finished processing](https://discuss.elastic.co/t/do-something-when-logstash-has-finished-processing/132729)
**Author:** [@saramali](https://discuss.elastic.co/u/saramali)\ **Replies:** 2\ **Last updated:** [May 22, 2018, 10:00am UTC](https://discuss.elastic.co/t/do-something-when-logstash-has-finished-processing/132729 "2018-05-22T10:00:10Z")
Is there any way I can know that logstash has finished processing a log file. I know logstash is meant to read data streams but is there a work around by which I can know logstash has finished processing of the file whic… --- ## [Custom pattern in grok in logstash](https://discuss.elastic.co/t/custom-pattern-in-grok-in-logstash/132761)
**Author:** [@aarthinim](https://discuss.elastic.co/u/aarthinim)\ **Replies:** 1\ **Last updated:** [May 22, 2018, 9:13am UTC](https://discuss.elastic.co/t/custom-pattern-in-grok-in-logstash/132761 "2018-05-22T09:13:10Z")
How to create custom pattern for this "(default task-612) " in logstash --- ## [Logstash not parsing logstash-tutorial.log](https://discuss.elastic.co/t/logstash-not-parsing-logstash-tutorial-log/132557)
**Author:** [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Replies:** 2\ **Last updated:** [May 22, 2018, 8:30am UTC](https://discuss.elastic.co/t/logstash-not-parsing-logstash-tutorial-log/132557 "2018-05-22T08:30:33Z")
Hello All, I have installed logstash and filebeat as per the tutorial on my centos 7 machine. I have followed the example at https://www.elastic.co/guide/en/logstash/6.2/advanced-pipeline.html However when I run it as … --- ## [Logstash date filter changing my time automatically](https://discuss.elastic.co/t/logstash-date-filter-changing-my-time-automatically/132743)
**Author:** [@devika\_sudheer](https://discuss.elastic.co/u/devika_sudheer)\ **Replies:** 1\ **Last updated:** [May 22, 2018, 8:00am UTC](https://discuss.elastic.co/t/logstash-date-filter-changing-my-time-automatically/132743 "2018-05-22T08:00:36Z")
My log data has a timestamp of the form Tue May 22 07:52:08 2018 825 I used logstash to convert this into date date { match =\> \["datetime", "EEE MMM dd HH:mm:ss yyyy SSS"\] target =\> "datetime" } But now the time s… --- ## [How to send csv file to logstash using filebeat](https://discuss.elastic.co/t/how-to-send-csv-file-to-logstash-using-filebeat/132334)
**Author:** [@Adansar](https://discuss.elastic.co/u/Adansar)\ **Replies:** 4\ **Last updated:** [May 22, 2018, 7:57am UTC](https://discuss.elastic.co/t/how-to-send-csv-file-to-logstash-using-filebeat/132334 "2018-05-22T07:57:33Z")
Hello, I'm trying to send csv data to logstash using filebeat, but i don't get the data on kibana. filebeat.yml paths: - C:\\ELK\\filebeat-5.6.2\\logs\*.csv - C:\\ELK\\filebeat-5.6.2\\logs\\test.log input\_type: log docume… --- ## [How can i use logstash to process custom log?](https://discuss.elastic.co/t/how-can-i-use-logstash-to-process-custom-log/132694)
**Author:** [@Anilkumar\_Patel](https://discuss.elastic.co/u/Anilkumar_Patel)\ **Replies:** 4\ **Last updated:** [May 22, 2018, 6:53am UTC](https://discuss.elastic.co/t/how-can-i-use-logstash-to-process-custom-log/132694 "2018-05-22T06:53:41Z")
Hi, I am new to the logstash and elasticsearch and I don't know how to process these below logs. fields in the log file are source, method, date & logmessage. Source : abcd.cs Method : abcd() Date : 15/05/… --- ## [OOM ElasticSearch OUTPUT](https://discuss.elastic.co/t/oom-elasticsearch-output/132158)
**Author:** [@aclowkey](https://discuss.elastic.co/u/aclowkey)\ **Replies:** 1\ **Last updated:** [May 22, 2018, 6:19am UTC](https://discuss.elastic.co/t/oom-elasticsearch-output/132158 "2018-05-22T06:19:55Z")
Using LogStash version 6.2.4. I have a rather complex pipeline that crashes when I use elasticsearch{}output, but works fine when I used stdout { codec =\> rubydebug}. There are around 115k document - also quite simple … --- ## [Logs bieng missed by logstash](https://discuss.elastic.co/t/logs-bieng-missed-by-logstash/132717)
**Author:** [@shaheryar](https://discuss.elastic.co/u/shaheryar)\ **Replies:** 3\ **Last updated:** [May 22, 2018, 5:44am UTC](https://discuss.elastic.co/t/logs-bieng-missed-by-logstash/132717 "2018-05-22T05:44:18Z")
I am collecting logs from a tcp input plugin but some logs get lost and there is no error. --- ## [Convert JSON key value to index](https://discuss.elastic.co/t/convert-json-key-value-to-index/132705)
**Author:** [@Stang](https://discuss.elastic.co/u/Stang)\ **Replies:** 1\ **Last updated:** [May 22, 2018, 5:01am UTC](https://discuss.elastic.co/t/convert-json-key-value-to-index/132705 "2018-05-22T05:01:00Z")
Hi, I am new to elk and have a usecase to solve, my input JSON has key value pairs nested with json, I have to be able to use one of these as indexes to report in Kibana, Did anyone solve this kind of issue? Any ideas o… --- ## [Unable to parse logs](https://discuss.elastic.co/t/unable-to-parse-logs/132654)
**Author:** [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Replies:** 4\ **Last updated:** [May 22, 2018, 4:50am UTC](https://discuss.elastic.co/t/unable-to-parse-logs/132654 "2018-05-22T04:50:11Z")
Hi, I have two different patterns in a single log file and I have two different groks handling them. But, somehow, the second grok is not able to parse the logs. I checked in grok debugger and everything is parsed corre… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=754) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=756)