# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=763 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 764 --- ## [NullPointerException in Cloudwatch Plugin](https://discuss.elastic.co/t/nullpointerexception-in-cloudwatch-plugin/131175)
**Author:** [@mikejr83](https://discuss.elastic.co/u/mikejr83)\ **Replies:** 0\ **Last updated:** [May 9, 2018, 1:26pm UTC](https://discuss.elastic.co/t/nullpointerexception-in-cloudwatch-plugin/131175 "2018-05-09T13:26:52Z")
I have a logstash instance running in AWS and it has a pipeline pulling from cloudwatch for metrics. This should be the standard cloudwatch input (https://github.com/logstash-plugins/logstash-input-cloudwatch). When I s… --- ## [Parse logs from log4j to logstash](https://discuss.elastic.co/t/parse-logs-from-log4j-to-logstash/131082)
**Author:** [@andradedb](https://discuss.elastic.co/u/andradedb)\ **Replies:** 3\ **Last updated:** [May 9, 2018, 1:19pm UTC](https://discuss.elastic.co/t/parse-logs-from-log4j-to-logstash/131082 "2018-05-09T13:19:46Z")
I'm trying to parse some old log files from log4j, but I can't find the right pattern for grok to do that. This is a snipet from the log 2018-04-12T15:10:25.180-0300 INFO CORE:VinculoServico:49 - teste 2018-04-12T15:1… --- ## [Elasticsearch output: Set user dynamically](https://discuss.elastic.co/t/elasticsearch-output-set-user-dynamically/131170)
**Author:** [@chp](https://discuss.elastic.co/u/chp)\ **Replies:** 1\ **Last updated:** [May 9, 2018, 1:13pm UTC](https://discuss.elastic.co/t/elasticsearch-output-set-user-dynamically/131170 "2018-05-09T13:13:26Z")
Hi, I would like to set the user of the elasticsearch output plugin dynamically based on the value of a field. Using the the syntax like %{user} doesn't work. Is there a way to achieve what I want? --- ## [Json-schema already in logstash?](https://discuss.elastic.co/t/json-schema-already-in-logstash/131172)
**Author:** [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Replies:** 0\ **Last updated:** [May 9, 2018, 12:56pm UTC](https://discuss.elastic.co/t/json-schema-already-in-logstash/131172 "2018-05-09T12:56:34Z")
I am writing a plugin to validate event data structure, hence I am using json-schema. When installing the gem inside the Docker elastic image, I got the error: logstash-filter-schema\_validation (= 0.1.0) java depend… --- ## [Get event data as a Ruby plain object](https://discuss.elastic.co/t/get-event-data-as-a-ruby-plain-object/131048)
**Author:** [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Replies:** 2\ **Last updated:** [May 9, 2018, 12:52pm UTC](https://discuss.elastic.co/t/get-event-data-as-a-ruby-plain-object/131048 "2018-05-09T12:52:33Z")
I am writing a plugin to validate the event data with JSON schema, with logstash 6.x, with https://github.com/ruby-json-schema/json-schema/. How can I get the event object as a plain ruby object? --- ## [Logstash 6.2.1 Big .since-db file causes OutOfMemory](https://discuss.elastic.co/t/logstash-6-2-1-big-since-db-file-causes-outofmemory/129501)
**Author:** [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)\ **Replies:** 21\ **Last updated:** [May 9, 2018, 12:23pm UTC](https://discuss.elastic.co/t/logstash-6-2-1-big-since-db-file-causes-outofmemory/129501 "2018-05-09T12:23:14Z")
Hi Team, I have a logstash instance that's been running for a while now but started falling over due to OutOfMemory error since yesterday. The memory is set as -Xms1g -Xmx1g. And it would continue to go out of memory … --- ## [Logstash 6.2.4 read\_to\_eof: no delimiter found in current chunk](https://discuss.elastic.co/t/logstash-6-2-4-read-to-eof-no-delimiter-found-in-current-chunk/130613)
**Author:** [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)\ **Replies:** 10\ **Last updated:** [May 9, 2018, 12:22pm UTC](https://discuss.elastic.co/t/logstash-6-2-4-read-to-eof-no-delimiter-found-in-current-chunk/130613 "2018-05-09T12:22:45Z")
Hi guys, I am using the new file input plugin 4.1.1 and keep getting the below warn message churning out numerous times - what is it about? \[WARN \]\[filewatch.tailmode.handlers.grow\] read\_to\_eof: no delimiter found in c… --- ## [File input v4.1.2 tailing - duplicate docs ingested](https://discuss.elastic.co/t/file-input-v4-1-2-tailing-duplicate-docs-ingested/130648)
**Author:** [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)\ **Replies:** 4\ **Last updated:** [May 9, 2018, 12:21pm UTC](https://discuss.elastic.co/t/file-input-v4-1-2-tailing-duplicate-docs-ingested/130648 "2018-05-09T12:21:47Z")
Hi Team, I'm seeing numerous duplicates (a lot of them) for each individual doc being sent by logstash. I am still unsure whether this is specific to 4.1.x plugin - I will a test with 4.0.3 and confirm this later. Belo… --- ## [Index logstash (Delete)](https://discuss.elastic.co/t/index-logstash-delete/130849)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 10\ **Last updated:** [May 9, 2018, 11:57am UTC](https://discuss.elastic.co/t/index-logstash-delete/130849 "2018-05-09T11:57:13Z")
Hello again, I delete logstash index with DELETE methode and I want to get it back, how can I do this ? --- ## [AWS Cloudwatch logs input plugin](https://discuss.elastic.co/t/aws-cloudwatch-logs-input-plugin/130777)
**Author:** [@rahulnama](https://discuss.elastic.co/u/rahulnama)\ **Replies:** 4\ **Last updated:** [May 9, 2018, 11:44am UTC](https://discuss.elastic.co/t/aws-cloudwatch-logs-input-plugin/130777 "2018-05-09T11:44:42Z")
Hello Team, I'm trying to pull Logs from AWS Cloudwatch. As you know Logstash input-cloudwatch plugin won't support to pull the logs. So I need to go with Community Developed Cloudwatch Plugin which pulls the Cloudwatc… --- ## [Grok filter with multiple regex not working](https://discuss.elastic.co/t/grok-filter-with-multiple-regex-not-working/131031)
**Author:** [@Dazith\_Kj](https://discuss.elastic.co/u/Dazith_Kj)\ **Replies:** 8\ **Last updated:** [May 9, 2018, 10:02am UTC](https://discuss.elastic.co/t/grok-filter-with-multiple-regex-not-working/131031 "2018-05-09T10:02:27Z")
I have design the below simple regex pattern. And this was working fine for me. input { beats { port =\> "5044" ssl =\> true ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt" … --- ## [Tracking any change to any of the multiple tables used in the join to create the index](https://discuss.elastic.co/t/tracking-any-change-to-any-of-the-multiple-tables-used-in-the-join-to-create-the-index/131007)
**Author:** [@RRSR](https://discuss.elastic.co/u/RRSR)\ **Replies:** 2\ **Last updated:** [May 9, 2018, 9:52am UTC](https://discuss.elastic.co/t/tracking-any-change-to-any-of-the-multiple-tables-used-in-the-join-to-create-the-index/131007 "2018-05-09T09:52:41Z")
Continuing the discussion from JDBC and tracking\_column(s) questions: I want to update the index whenever any of any of the tables used in the join to create the index is updated in MySQL. I'm using jdbc input plugin. … --- ## [Parse multiple lines of Log file into the same Elasticsearch document based on ID](https://discuss.elastic.co/t/parse-multiple-lines-of-log-file-into-the-same-elasticsearch-document-based-on-id/131145)
**Author:** [@bacem.hagui](https://discuss.elastic.co/u/bacem.hagui)\ **Replies:** 0\ **Last updated:** [May 9, 2018, 9:31am UTC](https://discuss.elastic.co/t/parse-multiple-lines-of-log-file-into-the-same-elasticsearch-document-based-on-id/131145 "2018-05-09T09:31:34Z")
I would like to store the times for different tasks of a job. The log file of my job has several lines each of which is a different task. Here is an example: 2018-04-25 18:33:40.940 INFO be8f587e-aae3-4e56-88a2-75dde556… --- ## [Different Kafka versions for Input and Output](https://discuss.elastic.co/t/different-kafka-versions-for-input-and-output/131059)
**Author:** [@gghdr](https://discuss.elastic.co/u/gghdr)\ **Replies:** 1\ **Last updated:** [May 9, 2018, 9:21am UTC](https://discuss.elastic.co/t/different-kafka-versions-for-input-and-output/131059 "2018-05-09T09:21:38Z")
Hi, like the title, my use case is consuming from Kafka 0.10.2.1 and producing to Kafka 0.9.0.0 The plugin versions i'm using are 6.3.4 for Input and 4.2.0 for Output. I'm able to consume but i'm having problems to pr… --- ## [Extract repeated fields, where number of fields varies](https://discuss.elastic.co/t/extract-repeated-fields-where-number-of-fields-varies/131127)
**Author:** [@vchandrashekar](https://discuss.elastic.co/u/vchandrashekar)\ **Replies:** 0\ **Last updated:** [May 9, 2018, 8:23am UTC](https://discuss.elastic.co/t/extract-repeated-fields-where-number-of-fields-varies/131127 "2018-05-09T08:23:01Z")
Hi All, I have a html log that gets generated from some of our servers. This is similar to below provided snippet. The key:value pairs for each operationName is operationName1 key1key1 related data here key2key2 rela… --- ## [Mutate --\> gsub - remove path and file extension from source](https://discuss.elastic.co/t/mutate-gsub-remove-path-and-file-extension-from-source/131119)
**Author:** [@dmccuk](https://discuss.elastic.co/u/dmccuk)\ **Replies:** 1\ **Last updated:** [May 9, 2018, 8:18am UTC](https://discuss.elastic.co/t/mutate-gsub-remove-path-and-file-extension-from-source/131119 "2018-05-09T08:18:58Z")
Hi, Is anyone able to point me in the right direction for this mutate --\> gsub I'm attempting. I have a source logfile with a path and file extension. What I'm looking to do is remove the path and the file extensions. … --- ## [Need help for parsing s3 logs to ELK dashboard](https://discuss.elastic.co/t/need-help-for-parsing-s3-logs-to-elk-dashboard/131120)
**Author:** [@Abdur\_Raqeeb1](https://discuss.elastic.co/u/Abdur_Raqeeb1)\ **Replies:** 0\ **Last updated:** [May 9, 2018, 7:51am UTC](https://discuss.elastic.co/t/need-help-for-parsing-s3-logs-to-elk-dashboard/131120 "2018-05-09T07:51:19Z")
please let us know how can i configure the logstash for multiple buckets and also please let me know for parsing the logs from multiple directories of buckets Below is a configuration file for particular bucket and pa… --- ## [Config Err](https://discuss.elastic.co/t/config-err/131077)
**Author:** [@naveensk](https://discuss.elastic.co/u/naveensk)\ **Replies:** 3\ **Last updated:** [May 9, 2018, 6:49am UTC](https://discuss.elastic.co/t/config-err/131077 "2018-05-09T06:49:51Z")
I just started working with ELK stack in my virtual box, below errors are not clear to me to debug. my csv file : 1,"pg100","dg101",2,"2016-01-01 00:29:24","2016-01-01 00:39:36",1 2,"pg100","dg2002",2,"2016-01-02 20:31… --- ## [Export Elastic search data to xls using Logstash](https://discuss.elastic.co/t/export-elastic-search-data-to-xls-using-logstash/130973)
**Author:** [@Swapnil.ch](https://discuss.elastic.co/u/Swapnil.ch)\ **Replies:** 2\ **Last updated:** [May 9, 2018, 6:49am UTC](https://discuss.elastic.co/t/export-elastic-search-data-to-xls-using-logstash/130973 "2018-05-09T06:49:09Z")
Hi, We have csv plugin to export data into csv using logstash. my requirement is to export elastic search data to xls format. Could you please help me to find anyt way to do that? --- ## [Cannot make Logstash read syslog](https://discuss.elastic.co/t/cannot-make-logstash-read-syslog/130687)
**Author:** [@AJ\_NOURI](https://discuss.elastic.co/u/AJ_NOURI)\ **Replies:** 9\ **Last updated:** [May 9, 2018, 3:03am UTC](https://discuss.elastic.co/t/cannot-make-logstash-read-syslog/130687 "2018-05-09T03:03:15Z")
Cannot make logstash receive syslog 514 traffic. ELK run successfully and listens to port 514: # docker-compose ps Name Command State Port… --- ## [There is no dateparsefailure in the tag,but it does not work](https://discuss.elastic.co/t/there-is-no-dateparsefailure-in-the-tag-but-it-does-not-work/130923)
**Author:** [@antony.y](https://discuss.elastic.co/u/antony.y)\ **Replies:** 9\ **Last updated:** [May 9, 2018, 2:59am UTC](https://discuss.elastic.co/t/there-is-no-dateparsefailure-in-the-tag-but-it-does-not-work/130923 "2018-05-09T02:59:58Z")
i have a field named "time",the format is just like "2018/05/02:08:22:23" , so i used a date filter, and the format is like following: date { match =\> \["time","yyyy/MM/dd:HH:mm:ss"\] targer =\>"@timestamp" } there is … --- ## [Logstash KV parsing : k=field, v=value](https://discuss.elastic.co/t/logstash-kv-parsing-k-field-v-value/131044)
**Author:** [@joaoantoniopereira](https://discuss.elastic.co/u/joaoantoniopereira)\ **Replies:** 4\ **Last updated:** [May 9, 2018, 12:52am UTC](https://discuss.elastic.co/t/logstash-kv-parsing-k-field-v-value/131044 "2018-05-09T00:52:26Z")
Hi, I am trying to use Logstash KV filter and I am having some difficulties. After some Logstash processing I get the following field "new\_parsed": "new\_parsed" =\> "{n=Tuner\_TotalHolePackets, v=0},{n=Tuner\_TotalPackets… --- ## [How can i use grok to match a single character of word](https://discuss.elastic.co/t/how-can-i-use-grok-to-match-a-single-character-of-word/130929)
**Author:** [@regex99](https://discuss.elastic.co/u/regex99)\ **Replies:** 3\ **Last updated:** [May 9, 2018, 12:23am UTC](https://discuss.elastic.co/t/how-can-i-use-grok-to-match-a-single-character-of-word/130929 "2018-05-09T00:23:35Z")
hi, I have a string like SAMPLE DATA 08X71 +F(9)L. i can match each word with %{WORD} like this, %{WORD:1}\\s\*%{WORD:2}\\s\*%{WORD} but how can I match each character separately from the last word in the same grok patte… --- ## [Multiple patterns in one index](https://discuss.elastic.co/t/multiple-patterns-in-one-index/131072)
**Author:** [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)\ **Replies:** 1\ **Last updated:** [May 8, 2018, 10:20pm UTC](https://discuss.elastic.co/t/multiple-patterns-in-one-index/131072 "2018-05-08T22:20:54Z")
\ 2018-02-07T10:42:02,831 \[ ExtractDwhData\] \[INFO \] Solife :: Tools :: DWH :: ITK DWH - version : 2.6.0-SNAPSHOT - build #265 on 2018-01-04 08:22:32 2018-02-07T10:42:02,832 \[ ExtractDwhData\] \[INFO \] Start… --- ## [Logstash gsub filter with special characters not working](https://discuss.elastic.co/t/logstash-gsub-filter-with-special-characters-not-working/131057)
**Author:** [@chakriv](https://discuss.elastic.co/u/chakriv)\ **Replies:** 2\ **Last updated:** [May 8, 2018, 9:42pm UTC](https://discuss.elastic.co/t/logstash-gsub-filter-with-special-characters-not-working/131057 "2018-05-08T21:42:37Z")
hello all, I am currently running logstash 6.x with ES 6.x and i am trying to leverage logstash to parse through the data and using mutate and gsub filters i have a syslog\_msg in the current format :slight\_smile: exam… --- ## [Issue with importing CVS file](https://discuss.elastic.co/t/issue-with-importing-cvs-file/130690)
**Author:** [@moe.kazemi](https://discuss.elastic.co/u/moe.kazemi)\ **Replies:** 5\ **Last updated:** [May 8, 2018, 8:07pm UTC](https://discuss.elastic.co/t/issue-with-importing-cvs-file/130690 "2018-05-08T20:07:29Z")
Hello all; I am using the following config. to import CVS file using logstash but when I call it, it give me the error that I have listed in item number 2. I would appreciate your assistance. Thanks item #1 input { … --- ## [Parsing Failed for multiline starts with dd MMM yyyy HH:mm:ss,SSS](https://discuss.elastic.co/t/parsing-failed-for-multiline-starts-with-dd-mmm-yyyy-hh-mm-ss-sss/130842)
**Author:** [@Anuj\_Rajput](https://discuss.elastic.co/u/Anuj_Rajput)\ **Replies:** 1\ **Last updated:** [May 8, 2018, 7:46pm UTC](https://discuss.elastic.co/t/parsing-failed-for-multiline-starts-with-dd-mmm-yyyy-hh-mm-ss-sss/130842 "2018-05-08T19:46:44Z")
Parsing Failed for multiline starts with dd MMM yyyy HH:mm:ss,SSS Sample log : 06 May 2018 18:25:01,021 Some Multiline Text Below configuration is used input { file { path =\> "/Some/Loactions/\*.log" type =\> "api\_ac… --- ## [Correct way to drop older events?](https://discuss.elastic.co/t/correct-way-to-drop-older-events/131043)
**Author:** [@jonrust](https://discuss.elastic.co/u/jonrust)\ **Replies:** 1\ **Last updated:** [May 8, 2018, 7:29pm UTC](https://discuss.elastic.co/t/correct-way-to-drop-older-events/131043 "2018-05-08T19:29:56Z")
I'd like to drop older events at logstash parsing time. Going with 100 days as my cutoff, I tried and failed to do this with the ruby module: ruby { code =\> "event.cancel if event.get(@timestamp) \< (Time.now - (864… --- ## [How use multiple if regex expression in output segment](https://discuss.elastic.co/t/how-use-multiple-if-regex-expression-in-output-segment/130991)
**Author:** [@adg](https://discuss.elastic.co/u/adg)\ **Replies:** 5\ **Last updated:** [May 8, 2018, 7:28pm UTC](https://discuss.elastic.co/t/how-use-multiple-if-regex-expression-in-output-segment/130991 "2018-05-08T19:28:16Z")
input input { beats { port =\> 5044 } } Filter csv{ separator =\> "," columns =\> \["DateTime" ,"hname" ,"uname" ,"cwd","fname","line"\] } Output if \[line\] in \["Delhi" , "Mumbai", "Chennai", "Kolkata" , "H… --- ## [TCP input and http output plugin configuration is not working](https://discuss.elastic.co/t/tcp-input-and-http-output-plugin-configuration-is-not-working/130907)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 5\ **Last updated:** [May 8, 2018, 7:24pm UTC](https://discuss.elastic.co/t/tcp-input-and-http-output-plugin-configuration-is-not-working/130907 "2018-05-08T19:24:49Z")
Hi, I am running logstash with tcp input and http output plugin(which is sending to another logstash from which it would be to ES) and I don't see any logs in ES. But when I replace tcp plugin with udp, I see logs in ES… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=762) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=764)