# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=575 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 576 --- ## [\[Resolved\]Logstash - Question for how to use if condition to skip invalid date format](https://discuss.elastic.co/t/resolved-logstash-question-for-how-to-use-if-condition-to-skip-invalid-date-format/174472)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 3\ **Last updated:** [April 1, 2019, 5:39am UTC](https://discuss.elastic.co/t/resolved-logstash-question-for-how-to-use-if-condition-to-skip-invalid-date-format/174472 "2019-04-01T05:39:07Z")
Hi Team, I have a CSV file which has a column named 'COMMIT\_DATE', the values of this column input by end user sometimes are a string , instead of a date. For example, there's invalid date value 'test' in below source d… --- ## [Grok for email log](https://discuss.elastic.co/t/grok-for-email-log/171750)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 13\ **Last updated:** [April 1, 2019, 4:35am UTC](https://discuss.elastic.co/t/grok-for-email-log/171750 "2019-04-01T04:35:05Z")
Can you please help me with building a grok pattern for SMTP log: Example of my smtp log: Return-Path: alok@XXXXXX.in Delivered-To: bigbrother@XXXX.in Received: from n-mail-1.xxxxx.in by n-mail-1.xxxxx.inwith LMTP i… --- ## [Logstash warn log:Attempted to resurrect connection to dead ES instance, but got an error](https://discuss.elastic.co/t/logstash-warn-log-attempted-to-resurrect-connection-to-dead-es-instance-but-got-an-error/174701)
**Author:** [@ly217](https://discuss.elastic.co/u/ly217)\ **Replies:** 0\ **Last updated:** [April 1, 2019, 2:46am UTC](https://discuss.elastic.co/t/logstash-warn-log-attempted-to-resurrect-connection-to-dead-es-instance-but-got-an-error/174701 "2019-04-01T02:46:46Z")
The logstash always logging this warn log.I want some help . \[2019-04-01T10:42:20,243\]\[WARN \]\[logstash.outputs.elasticsearch\] Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>"http://es-35… --- ## [Is there a functionality like before insert available](https://discuss.elastic.co/t/is-there-a-functionality-like-before-insert-available/174700)
**Author:** [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\ **Replies:** 0\ **Last updated:** [April 1, 2019, 2:29am UTC](https://discuss.elastic.co/t/is-there-a-functionality-like-before-insert-available/174700 "2019-04-01T02:29:32Z")
Hello Team, Is it possible to check if document with matching query is already available in the index? if yes use the indexed field value to set field in new document. If match is not found create a field with default v… --- ## [Logstash plugin install error](https://discuss.elastic.co/t/logstash-plugin-install-error/174695)
**Author:** [@MagicKim](https://discuss.elastic.co/u/MagicKim)\ **Replies:** 0\ **Last updated:** [April 1, 2019, 12:29am UTC](https://discuss.elastic.co/t/logstash-plugin-install-error/174695 "2019-04-01T00:29:07Z")
Hello I just installed ELK on Windows Server 2012 R2 And I need to beats plugin, so I did it on cmd like that After that, I set JAVA\_OPTS and did it again. But, I received error again. What should I do? --- ## [Logstash elastic search templete error](https://discuss.elastic.co/t/logstash-elastic-search-templete-error/173404)
**Author:** [@Lokesh\_Reddy](https://discuss.elastic.co/u/Lokesh_Reddy)\ **Replies:** 0\ **Last updated:** [March 21, 2019, 9:31pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-templete-error/173404 "2019-03-21T21:31:47Z")
Hello, I am collecting data from twitter, I have found something online and I was keep on trying to get twitter data into elastic search finally I have found we need to use the templete. I have used this templete { … --- ## [Kv filter parsing](https://discuss.elastic.co/t/kv-filter-parsing/174674)
**Author:** [@paano](https://discuss.elastic.co/u/paano)\ **Replies:** 4\ **Last updated:** [March 31, 2019, 6:06pm UTC](https://discuss.elastic.co/t/kv-filter-parsing/174674 "2019-03-31T18:06:28Z")
Is there a way to use KV filter for parse payload patterns as below: A(1), B(2), C(3), D(T1) Looked into config-option include\_brackets, trim\_key, trim\_value logstash filter is as below: filter { kv { value\_split… --- ## [Formatting output of report for Logstash / Elasticsearch](https://discuss.elastic.co/t/formatting-output-of-report-for-logstash-elasticsearch/174658)
**Author:** [@createchange](https://discuss.elastic.co/u/createchange)\ **Replies:** 1\ **Last updated:** [March 31, 2019, 3:49pm UTC](https://discuss.elastic.co/t/formatting-output-of-report-for-logstash-elasticsearch/174658 "2019-03-31T15:49:16Z")
Hey there. I have been tasked with writing a report to obtain data re: cpanel account sites (think disk and bandwidth utilization, account packages, etc.). This started me down the path of the Elastic Stack, which I have… --- ## [Lowercase all fields and sub fields](https://discuss.elastic.co/t/lowercase-all-fields-and-sub-fields/174438)
**Author:** [@tellus83](https://discuss.elastic.co/u/tellus83)\ **Replies:** 1\ **Last updated:** [March 30, 2019, 9:26pm UTC](https://discuss.elastic.co/t/lowercase-all-fields-and-sub-fields/174438 "2019-03-30T21:26:10Z")
I'm trying to convert all fields name to lowercase. This example converts the first fields name but need to convert all sub fields aswell. ruby { code =\> " event.to\_hash.keys.each do |y| event.s… --- ## [Robust filter to categorise all logs into "log" and "message"](https://discuss.elastic.co/t/robust-filter-to-categorise-all-logs-into-log-and-message/174669)
**Author:** [@igor1](https://discuss.elastic.co/u/igor1)\ **Replies:** 0\ **Last updated:** [March 31, 2019, 11:31am UTC](https://discuss.elastic.co/t/robust-filter-to-categorise-all-logs-into-log-and-message/174669 "2019-03-31T11:31:14Z")
Hello there! I would like to upgrade our ELK pipeline from 5.6.6 to 6.6.2 I had some problems with ES and Kibana which by now were successfully solved and I finally got to Logstash and Filebeat. In v5.6.6 we used quit… --- ## [Drop/cancel events with a timestamp older than a given amount of seconds](https://discuss.elastic.co/t/drop-cancel-events-with-a-timestamp-older-than-a-given-amount-of-seconds/174608)
**Author:** [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Replies:** 3\ **Last updated:** [March 30, 2019, 7:40pm UTC](https://discuss.elastic.co/t/drop-cancel-events-with-a-timestamp-older-than-a-given-amount-of-seconds/174608 "2019-03-30T19:40:55Z")
LogStash 6.6 My log files have lines like this 01/07/19 18:31:33.452 blah blah blah where the time is local. I need to discard all lines older than a given amount of time. Let's say, for example, 24 hours. I have a … --- ## [How filter Cloudwatch Logs Aws?](https://discuss.elastic.co/t/how-filter-cloudwatch-logs-aws/174638)
**Author:** [@angelicamaria](https://discuss.elastic.co/u/angelicamaria)\ **Replies:** 3\ **Last updated:** [March 30, 2019, 3:46pm UTC](https://discuss.elastic.co/t/how-filter-cloudwatch-logs-aws/174638 "2019-03-30T15:46:37Z")
Hello guys, How do I make the filter take into account the records that are between START and END? I want to send to Elastic something as this: { "\_index": "pruebalog:07132017", "\_type": "cloudwatch\_logs", "\_id… --- ## [Logstash via kafka - case for pipeline usage?](https://discuss.elastic.co/t/logstash-via-kafka-case-for-pipeline-usage/174637)
**Author:** [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Replies:** 1\ **Last updated:** [March 30, 2019, 3:24pm UTC](https://discuss.elastic.co/t/logstash-via-kafka-case-for-pipeline-usage/174637 "2019-03-30T15:24:12Z")
Loading metric/file beat data via 2 kafka queues. Consumer are configured to eacI have a single 'filter' file with 20-30 index choices determined via if/then/else if. Could I get better performance if I split this up to … --- ## [Grokparsefailure on COMBINEDAPACHELOG not matching empty USER with quotes](https://discuss.elastic.co/t/grokparsefailure-on-combinedapachelog-not-matching-empty-user-with-quotes/174636)
**Author:** [@stasher](https://discuss.elastic.co/u/stasher)\ **Replies:** 0\ **Last updated:** [March 30, 2019, 12:59pm UTC](https://discuss.elastic.co/t/grokparsefailure-on-combinedapachelog-not-matching-empty-user-with-quotes/174636 "2019-03-30T12:59:02Z")
I have noticed that this line in access.log is getting a \_grokparsefailure : 220.189.249.66 - "" \[18/Mar/2019:13:24:42 +0000\] "GET /manager/html HTTP/1.1" 200 2756 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2… --- ## [How to replace or remove hyphen (-) in a string field](https://discuss.elastic.co/t/how-to-replace-or-remove-hyphen-in-a-string-field/174633)
**Author:** [@jwjhou](https://discuss.elastic.co/u/jwjhou)\ **Replies:** 1\ **Last updated:** [March 30, 2019, 12:40pm UTC](https://discuss.elastic.co/t/how-to-replace-or-remove-hyphen-in-a-string-field/174633 "2019-03-30T12:40:27Z")
I have a string field called log\_type, which may contains string like "-- INFO". I'd like to remove hyphens (i. e. replace hyphen with empty string), how to do it? I have tried mutate gsub, like this: mutate { gsu… --- ## [Logstash keeps restarting after last upgrade to v6.7](https://discuss.elastic.co/t/logstash-keeps-restarting-after-last-upgrade-to-v6-7/174614)
**Author:** [@uhlhosting](https://discuss.elastic.co/u/uhlhosting)\ **Replies:** 0\ **Last updated:** [March 29, 2019, 10:17pm UTC](https://discuss.elastic.co/t/logstash-keeps-restarting-after-last-upgrade-to-v6-7/174614 "2019-03-29T22:17:38Z")
After a while decided to manually remove all plugins and modules, and reinstalled elasticsearch, this helped, while meanwhile, logstash fails with: \[2019-03-29T23:02:47,796\]\[ERROR\]\[org.logstash.Logstash \] java.lang.I… --- ## [Logstash filtering](https://discuss.elastic.co/t/logstash-filtering/174570)
**Author:** [@Elk\_huh](https://discuss.elastic.co/u/Elk_huh)\ **Replies:** 8\ **Last updated:** [March 29, 2019, 6:51pm UTC](https://discuss.elastic.co/t/logstash-filtering/174570 "2019-03-29T18:51:54Z")
How do you filter something that has multiple of the same Keys. in my "message" i have 3 different \_id keys. im currently using kv to split up my data , but it is showing all 3 values into that 1 key Its not working … --- ## [Painless script to add value to array if its exists](https://discuss.elastic.co/t/painless-script-to-add-value-to-array-if-its-exists/174580)
**Author:** [@nicolas\_b](https://discuss.elastic.co/u/nicolas_b)\ **Replies:** 2\ **Last updated:** [March 29, 2019, 6:43pm UTC](https://discuss.elastic.co/t/painless-script-to-add-value-to-array-if-its-exists/174580 "2019-03-29T18:43:49Z")
Hello, i have an input with ( besides other data) 2 Arrays , event\_type and duration\_ms. i use logstash to update the document with this data. Because i don't want to loose the data from event\_type and duration, i just … --- ## [Logstash output as CEF data to other server](https://discuss.elastic.co/t/logstash-output-as-cef-data-to-other-server/174456)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 3\ **Last updated:** [March 29, 2019, 6:08pm UTC](https://discuss.elastic.co/t/logstash-output-as-cef-data-to-other-server/174456 "2019-03-29T18:08:01Z")
Hi there, I have elastic search stack where logstash is accepting data and ingesting in elastic search now I need to have another output configured and forward the data to a third part collector which is accepting in CE… --- ## [Logstash 6.7.0 on Raspberry Pi 3 B+ - java.lang.IllegalStateException](https://discuss.elastic.co/t/logstash-6-7-0-on-raspberry-pi-3-b-java-lang-illegalstateexception/174588)
**Author:** [@James\_DeHaan](https://discuss.elastic.co/u/James_DeHaan)\ **Replies:** 0\ **Last updated:** [March 29, 2019, 5:53pm UTC](https://discuss.elastic.co/t/logstash-6-7-0-on-raspberry-pi-3-b-java-lang-illegalstateexception/174588 "2019-03-29T17:53:38Z")
Howdy All Quick background, I had tried to get this going: https:// github.com/TravisFSmith/SweetSecurity but removed LS 5.5.X completely and installed LS 6.7.0 via dpkg/logstash-6.7.0.deb Version: logstash-6.7.0 … --- ## [Using ILM with NetFlow module](https://discuss.elastic.co/t/using-ilm-with-netflow-module/173969)
**Author:** [@Ryne\_Keel](https://discuss.elastic.co/u/Ryne_Keel)\ **Replies:** 6\ **Last updated:** [March 29, 2019, 3:53pm UTC](https://discuss.elastic.co/t/using-ilm-with-netflow-module/173969 "2019-03-29T15:53:25Z")
This is hard to explain, but index life cycle policies do not work with at least the logstash netflow module. The module, or elasticsearch create a new index each day. They use the index template to do so. To get it to … --- ## [If statement to check if difference between two values is greater than number](https://discuss.elastic.co/t/if-statement-to-check-if-difference-between-two-values-is-greater-than-number/174432)
**Author:** [@greengeek](https://discuss.elastic.co/u/greengeek)\ **Replies:** 5\ **Last updated:** [March 29, 2019, 5:06pm UTC](https://discuss.elastic.co/t/if-statement-to-check-if-difference-between-two-values-is-greater-than-number/174432 "2019-03-29T17:06:03Z")
I have two fields that are epoch timestamps. I want to take the first field and subtract the second field. If the results are larger than a certain number, I want to write a value into a new field. Basically an IF sta… --- ## [Correct syntax to my filter](https://discuss.elastic.co/t/correct-syntax-to-my-filter/174562)
**Author:** [@espacioint](https://discuss.elastic.co/u/espacioint)\ **Replies:** 3\ **Last updated:** [March 29, 2019, 4:10pm UTC](https://discuss.elastic.co/t/correct-syntax-to-my-filter/174562 "2019-03-29T16:10:33Z")
Hello,my name is Pablo, this is my first post so I ask for a bit a patience about me.. im new in elk. I'm working with netflow. I have many fields that Im not interested to store, so I can have less data on my elk. I… --- ## [Initial setup instructions: how to take the timestamp from the log?](https://discuss.elastic.co/t/initial-setup-instructions-how-to-take-the-timestamp-from-the-log/174554)
**Author:** [@stasher](https://discuss.elastic.co/u/stasher)\ **Replies:** 0\ **Last updated:** [March 29, 2019, 3:03pm UTC](https://discuss.elastic.co/t/initial-setup-instructions-how-to-take-the-timestamp-from-the-log/174554 "2019-03-29T15:03:51Z")
First time trying this out, and I have successfully got through the excellent Getting Started instructions. It is working. Some feedback and a question: It did seem a bit odd that all the datapoints had the same timest… --- ## [Logstash sum values from jdbc input](https://discuss.elastic.co/t/logstash-sum-values-from-jdbc-input/174393)
**Author:** [@Thiago\_Anate](https://discuss.elastic.co/u/Thiago_Anate)\ **Replies:** 3\ **Last updated:** [March 29, 2019, 2:50pm UTC](https://discuss.elastic.co/t/logstash-sum-values-from-jdbc-input/174393 "2019-03-29T14:50:42Z")
Hello, I'm new on logstash and i'm trying to sum the value from two colums from my database and generate a new metric. I've exhausted all my alternatives. this is my conf file. the new varible that I'm creating is the… --- ## [Logstash docker pipeline volume not mounting](https://discuss.elastic.co/t/logstash-docker-pipeline-volume-not-mounting/173642)
**Author:** [@iEMH](https://discuss.elastic.co/u/iEMH)\ **Replies:** 1\ **Last updated:** [March 29, 2019, 1:10pm UTC](https://discuss.elastic.co/t/logstash-docker-pipeline-volume-not-mounting/173642 "2019-03-29T13:10:28Z")
Hi, I am hoping someone can help me spot where am I doing a mistake. I am trying to spin up a simple logstash docker container (oss) with a persistent volume mounted to the default pipeline directory. However, when I … --- ## [Error: Could not find or load main class Files - Logstash installation failed](https://discuss.elastic.co/t/error-could-not-find-or-load-main-class-files-logstash-installation-failed/174473)
**Author:** [@Sikha\_Arun](https://discuss.elastic.co/u/Sikha_Arun)\ **Replies:** 3\ **Last updated:** [March 29, 2019, 1:02pm UTC](https://discuss.elastic.co/t/error-could-not-find-or-load-main-class-files-logstash-installation-failed/174473 "2019-03-29T13:02:47Z")
Cannot install logstash on my windows 10 system . Error: Could not find or load main class Files Caused by: java.lang.ClassNotFoundException: Files Logstash.conf file is as follows JAVA\_HOME environment variables… --- ## [Grok pattern works at debugger but filter](https://discuss.elastic.co/t/grok-pattern-works-at-debugger-but-filter/174261)
**Author:** [@gray380](https://discuss.elastic.co/u/gray380)\ **Replies:** 2\ **Last updated:** [March 29, 2019, 11:52am UTC](https://discuss.elastic.co/t/grok-pattern-works-at-debugger-but-filter/174261 "2019-03-29T11:52:34Z")
Hello, The following strings successfully parsed at grok debugger: \<174\>ddsh: {epoch=1553696401;id='AUDIT-DDSH-00001';desc='DDSH CLI command';level=3;user='root';role='se';app='ddsh';host='ddem';detail='cmd=cifs show d… --- ## [High cpu usage on Logstash Netflow receiver](https://discuss.elastic.co/t/high-cpu-usage-on-logstash-netflow-receiver/174522)
**Author:** [@valentin2105](https://discuss.elastic.co/u/valentin2105)\ **Replies:** 0\ **Last updated:** [March 29, 2019, 11:07am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-logstash-netflow-receiver/174522 "2019-03-29T11:07:06Z")
Hi, It's my first setup of Logstash, I already have a 3 nodes ES cluster from a while, which receive Filebeat / Metricbeat datas. I setup my Logstash server to receive Netflow (Small ISP size network w/ 6k customers),… --- ## [Logstash stopping and starting, is this normal behaviour?](https://discuss.elastic.co/t/logstash-stopping-and-starting-is-this-normal-behaviour/173509)
**Author:** [@Jeremy\_A](https://discuss.elastic.co/u/Jeremy_A)\ **Replies:** 3\ **Last updated:** [March 29, 2019, 10:57am UTC](https://discuss.elastic.co/t/logstash-stopping-and-starting-is-this-normal-behaviour/173509 "2019-03-29T10:57:50Z")
I have setup Logstash to take input from one ES index based on a query and then output to another ES index. It is working, but going very slowly, and from the logs it seems to be stopping and starting every few seconds. … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=574) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=576)