# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=958 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 959 --- ## [Parsing mail from (AWS) S3](https://discuss.elastic.co/t/parsing-mail-from-aws-s3/90779)
**Author:** [@Jackson\_Pollock](https://discuss.elastic.co/u/Jackson_Pollock)\ **Replies:** 1\ **Last updated:** [June 26, 2017, 8:47am UTC](https://discuss.elastic.co/t/parsing-mail-from-aws-s3/90779 "2017-06-26T08:47:07Z")
I have email (text files) in s3 that is being forwarded via SES. These are simple text files with the contents of a single email per file. My logstash below: input { s3 { access\_key\_id =\> "xxxxxxx" secret\_acc… --- ## [Index auto increment issue](https://discuss.elastic.co/t/index-auto-increment-issue/87040)
**Author:** [@kamran.matloob](https://discuss.elastic.co/u/kamran.matloob)\ **Replies:** 7\ **Last updated:** [June 26, 2017, 8:39am UTC](https://discuss.elastic.co/t/index-auto-increment-issue/87040 "2017-06-26T08:39:30Z")
My requirement is to rollover indices after 7days. I successfully created dynamic index in my logstsh.conf file by using below link https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-aliases.html Below I… --- ## [Not able to Run Logstash .Conf file in ELK 5.4 (Current) Version](https://discuss.elastic.co/t/not-able-to-run-logstash-conf-file-in-elk-5-4-current-version/90476)
**Author:** [@Pandiyan\_M](https://discuss.elastic.co/u/Pandiyan_M)\ **Replies:** 4\ **Last updated:** [June 26, 2017, 7:23am UTC](https://discuss.elastic.co/t/not-able-to-run-logstash-conf-file-in-elk-5-4-current-version/90476 "2017-06-26T07:23:57Z")
Hi, I have prepared logstash.conf under/etc/logstash/conf.d/filelog.conf as below input { file { path =\> "/var/log/file.log" codec =\> json start\_position =\> "beginning" } } filter{ grok { … --- ## [Vendor MIBs for snmptrap module -\> import.rb fails every time](https://discuss.elastic.co/t/vendor-mibs-for-snmptrap-module-import-rb-fails-every-time/90788)
**Author:** [@MarcusCaepio](https://discuss.elastic.co/u/MarcusCaepio)\ **Replies:** 0\ **Last updated:** [June 26, 2017, 7:21am UTC](https://discuss.elastic.co/t/vendor-mibs-for-snmptrap-module-import-rb-fails-every-time/90788 "2017-06-26T07:21:21Z")
Hi All, so I am new to Elastick Stack and new to snmptrap plugin. Anyway, my pipeline works with the snmptrap listener, but the default MIBs are not complete. So I wanted to import my vendors MIBs. Doesn't matter which M… --- ## [Split arrays of keys and values](https://discuss.elastic.co/t/split-arrays-of-keys-and-values/90736)
**Author:** [@Khaled](https://discuss.elastic.co/u/Khaled)\ **Replies:** 3\ **Last updated:** [June 26, 2017, 5:58am UTC](https://discuss.elastic.co/t/split-arrays-of-keys-and-values/90736 "2017-06-26T05:58:31Z")
I'm using Amazon SQS input plugin to get bounced & delivery reports from Amazon SES (Simple Email Service). One of the fields (mail.headers) is an array of keys and values, like the following: { "name": "Message… --- ## [Multiple problems with logstash](https://discuss.elastic.co/t/multiple-problems-with-logstash/90554)
**Author:** [@sfunk](https://discuss.elastic.co/u/sfunk)\ **Replies:** 1\ **Last updated:** [June 26, 2017, 5:53am UTC](https://discuss.elastic.co/t/multiple-problems-with-logstash/90554 "2017-06-26T05:53:34Z")
I'm encountering a couple issues with logstash, it's probably me not understanding the documentation. First, I cannot get logstash to start where it left off when filling ES from an Oracle datastore. When it gets about… --- ## [How to use grok filter on JSON field](https://discuss.elastic.co/t/how-to-use-grok-filter-on-json-field/90578)
**Author:** [@sachiiit](https://discuss.elastic.co/u/sachiiit)\ **Replies:** 1\ **Last updated:** [June 26, 2017, 5:34am UTC](https://discuss.elastic.co/t/how-to-use-grok-filter-on-json-field/90578 "2017-06-26T05:34:56Z")
I have an http input plugin in my logstash configuration with codec as json as http { host =\> "0.0.0.0" port =\> "8080" user =\> "username" password =\> "… --- ## [Docker.elastic.co/logstash/logstash:5.4.2 failed to start](https://discuss.elastic.co/t/docker-elastic-co-logstash-logstash-5-4-2-failed-to-start/90633)
**Author:** [@vzh](https://discuss.elastic.co/u/vzh)\ **Replies:** 0\ **Last updated:** [June 23, 2017, 12:21pm UTC](https://discuss.elastic.co/t/docker-elastic-co-logstash-logstash-5-4-2-failed-to-start/90633 "2017-06-23T12:21:48Z")
Hello. When I try to run next service: docker service create --name logstash \\ --mount "type=bind,source=$PWD/docker/logstash/logstash.conf,target=/usr/share/logstash/pipeline/logstash.conf" \\ --mount "type=bind,so… --- ## [Logstash log4j warning](https://discuss.elastic.co/t/logstash-log4j-warning/90547)
**Author:** [@pashmina](https://discuss.elastic.co/u/pashmina)\ **Replies:** 1\ **Last updated:** [June 26, 2017, 5:20am UTC](https://discuss.elastic.co/t/logstash-log4j-warning/90547 "2017-06-26T05:20:49Z")
I am using java version "1.8.0\_74" logstash 5.4.2 In logstash-stderr.log , I see the following error. log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAuthCache). log4j:WARN … --- ## [A plugin had an unrecoverable error. Will restart this plugin Error: No such file or directory](https://discuss.elastic.co/t/a-plugin-had-an-unrecoverable-error-will-restart-this-plugin-error-no-such-file-or-directory/90467)
**Author:** [@kalepu.girish](https://discuss.elastic.co/u/kalepu.girish)\ **Replies:** 1\ **Last updated:** [June 26, 2017, 3:33am UTC](https://discuss.elastic.co/t/a-plugin-had-an-unrecoverable-error-will-restart-this-plugin-error-no-such-file-or-directory/90467 "2017-06-26T03:33:57Z")
I am getting the below error , could you please help , i cant create the home directory on my server . Is there a config somewhere we can change to desired location A plugin had an unrecoverable error. Will restart thi… --- ## [Mysql error grok problem](https://discuss.elastic.co/t/mysql-error-grok-problem/90764)
**Author:** [@Pietia](https://discuss.elastic.co/u/Pietia)\ **Replies:** 0\ **Last updated:** [June 25, 2017, 6:23pm UTC](https://discuss.elastic.co/t/mysql-error-grok-problem/90764 "2017-06-25T18:23:23Z")
Hello, I'm trying to parse following log entries: 2017-06-25 20:06:36 139710940705344 \[Note\] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.35-80.0 started; log sequence number 1616995 2017-06-25 20:06:36 1397109… --- ## [Parse catalina error log](https://discuss.elastic.co/t/parse-catalina-error-log/90730)
**Author:** [@Pietia](https://discuss.elastic.co/u/Pietia)\ **Replies:** 0\ **Last updated:** [June 24, 2017, 9:49pm UTC](https://discuss.elastic.co/t/parse-catalina-error-log/90730 "2017-06-24T21:49:17Z")
problem is solved it was beats sending messages over to elastic directly . --- ## [Duplicate message in nginx logs](https://discuss.elastic.co/t/duplicate-message-in-nginx-logs/90757)
**Author:** [@Pietia](https://discuss.elastic.co/u/Pietia)\ **Replies:** 1\ **Last updated:** [June 25, 2017, 3:12pm UTC](https://discuss.elastic.co/t/duplicate-message-in-nginx-logs/90757 "2017-06-25T15:12:05Z")
problem is solved it was beats sending messages over to elastic directly . --- ## [Logstash Kafka input loosing events](https://discuss.elastic.co/t/logstash-kafka-input-loosing-events/90746)
**Author:** [@guy\_es](https://discuss.elastic.co/u/guy_es)\ **Replies:** 0\ **Last updated:** [June 25, 2017, 7:38am UTC](https://discuss.elastic.co/t/logstash-kafka-input-loosing-events/90746 "2017-06-25T07:38:17Z")
Hi All, I use logstash 2.4.1 with logstash-input-kafka=5.1.6 and output to ES 2.4, kafka 0.10, 160 partitions. I insert 1000 events to a kafka topic and only about half of the events are being written to ES, I don't se… --- ## [Is there a way to add "userdata" to an input-definition?](https://discuss.elastic.co/t/is-there-a-way-to-add-userdata-to-an-input-definition/90674)
**Author:** [@UnitedMarsupials](https://discuss.elastic.co/u/UnitedMarsupials)\ **Replies:** 3\ **Last updated:** [June 24, 2017, 7:14pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-add-userdata-to-an-input-definition/90674 "2017-06-24T19:14:25Z")
I'd like the data read from a particular input to be sent to a different index in ElasticSearch. I tried the following, but logstash refuses it for syntax error -- setting @metadata in an input-definition is, apparently,… --- ## [Speeding up Logstash using Docker](https://discuss.elastic.co/t/speeding-up-logstash-using-docker/90643)
**Author:** [@CDR](https://discuss.elastic.co/u/CDR)\ **Replies:** 1\ **Last updated:** [June 24, 2017, 7:12pm UTC](https://discuss.elastic.co/t/speeding-up-logstash-using-docker/90643 "2017-06-24T19:12:23Z")
Hello, I am currently using Docker to run the ELK stack. I have used Vagrant and a dedicated as two other methods to run the stack. Both of those have allowed for fairly quick parsing of logs. I am investigating as to … --- ## [How do you store variables for logstash?](https://discuss.elastic.co/t/how-do-you-store-variables-for-logstash/90654)
**Author:** [@rcanzanese](https://discuss.elastic.co/u/rcanzanese)\ **Replies:** 1\ **Last updated:** [June 24, 2017, 7:10pm UTC](https://discuss.elastic.co/t/how-do-you-store-variables-for-logstash/90654 "2017-06-24T19:10:08Z")
There are a few things that I am doing in logstash for which I would like to store variables in a file that is separate from the logstash filter configs themselves. For example, I am using the CIDR filter in a few diffe… --- ## [Use saved saved search template with http\_poller](https://discuss.elastic.co/t/use-saved-saved-search-template-with-http-poller/90324)
**Author:** [@adonahoe](https://discuss.elastic.co/u/adonahoe)\ **Replies:** 1\ **Last updated:** [June 24, 2017, 7:07pm UTC](https://discuss.elastic.co/t/use-saved-saved-search-template-with-http-poller/90324 "2017-06-24T19:07:12Z")
Hi, I am attempting to call a saved search with http\_poller and currently cannot figure it out. Was wondering if any others have run into this issue before? My config is below: Search Template: PUT \_search/te… --- ## [Logstash IRC output plugin creates new IRC server connections on reload](https://discuss.elastic.co/t/logstash-irc-output-plugin-creates-new-irc-server-connections-on-reload/90293)
**Author:** [@bjorn](https://discuss.elastic.co/u/bjorn)\ **Replies:** 1\ **Last updated:** [June 24, 2017, 7:06pm UTC](https://discuss.elastic.co/t/logstash-irc-output-plugin-creates-new-irc-server-connections-on-reload/90293 "2017-06-24T19:06:35Z")
Using the IRC output plugin and automatic config reload (config.reload.automatic: true in logstash.yml), Logstash will establish a brand new connection to the IRC server when config changes. The new connections will have… --- ## [exception=\>#\ **Author:** [@vikas\_gopal](https://discuss.elastic.co/u/vikas_gopal)\ **Replies:** 1\ **Last updated:** [June 24, 2017, 6:53pm UTC](https://discuss.elastic.co/t/exception-csv-malformedcsverror-unclosed-quoted-field-on-line-1/90703 "2017-06-24T18:53:33Z") Hi Experts, I am getting exception=\>#\ **Author:** [@Dorian\_C](https://discuss.elastic.co/u/Dorian_C)\ **Replies:** 0\ **Last updated:** [June 24, 2017, 4:28pm UTC](https://discuss.elastic.co/t/setting-type-within-logstash-kv/90718 "2017-06-24T16:28:48Z") Hello, I'm having trouble to determine the type of dynamically created fields with kv filter. My config looks like this: input { beats { port =\> 5044 } } filter { grok { match =\> { "message" =\> "%{COMBI… --- ## [Can't create nested type in ElasticSearch from data out of Logstash](https://discuss.elastic.co/t/cant-create-nested-type-in-elasticsearch-from-data-out-of-logstash/89585)
**Author:** [@Sesam](https://discuss.elastic.co/u/Sesam)\ **Replies:** 1\ **Last updated:** [June 15, 2017, 4:46pm UTC](https://discuss.elastic.co/t/cant-create-nested-type-in-elasticsearch-from-data-out-of-logstash/89585 "2017-06-15T16:46:04Z")
Hey guys, im suffering with this Problem: Using jdbc connected with SQL-Database. I want to achive this nested type in elasticsearch: { "mappings": { "Units": { "properties": { … --- ## [Multiple pipe inputs supported?](https://discuss.elastic.co/t/multiple-pipe-inputs-supported/90688)
**Author:** [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Replies:** 1\ **Last updated:** [June 24, 2017, 5:56am UTC](https://discuss.elastic.co/t/multiple-pipe-inputs-supported/90688 "2017-06-24T05:56:25Z")
Hi All! is the following configuration supported? input { pipe{ command =\> "/usr/bin/stdbuf -oL /some/shell/script param1" type =\> "type1" } pipe{ command =\> "/usr/bin/stdbuf -oL… --- ## [Winlogbeat via Logstash into Elasticsearch](https://discuss.elastic.co/t/winlogbeat-via-logstash-into-elasticsearch/90081)
**Author:** [@hispeed](https://discuss.elastic.co/u/hispeed)\ **Replies:** 17\ **Last updated:** [June 24, 2017, 5:40am UTC](https://discuss.elastic.co/t/winlogbeat-via-logstash-into-elasticsearch/90081 "2017-06-24T05:40:32Z")
Hi, I have a on a Windows machine winlogbeat and it's working fine so far. At the moment all logs are going into the same index in elasticsearch. Yes i have searched and found topics with that question but i was unable … --- ## [Logstash nlp error cannot create pipeline](https://discuss.elastic.co/t/logstash-nlp-error-cannot-create-pipeline/90432)
**Author:** [@Nusrath](https://discuss.elastic.co/u/Nusrath)\ **Replies:** 6\ **Last updated:** [June 24, 2017, 5:18am UTC](https://discuss.elastic.co/t/logstash-nlp-error-cannot-create-pipeline/90432 "2017-06-24T05:18:32Z")
http://blog.jaywayco.co.uk/analyze-sentiment-and-log-all-the-things/ i have used this blog ,and followed the steps to download NLP jars. my logstash.conf file is look like as follow input { stdin {} } filter { … --- ## [\[Elapsed Plugin\] Is it possible to add TAG to start event when end event match?](https://discuss.elastic.co/t/elapsed-plugin-is-it-possible-to-add-tag-to-start-event-when-end-event-match/90690)
**Author:** [@daian.martinho](https://discuss.elastic.co/u/daian.martinho)\ **Replies:** 0\ **Last updated:** [June 24, 2017, 12:20am UTC](https://discuss.elastic.co/t/elapsed-plugin-is-it-possible-to-add-tag-to-start-event-when-end-event-match/90690 "2017-06-24T00:20:21Z")
I'm using elapsed plugin. I have start event, and end event. if \[trigger\_status\] == "PROBLEM" { mutate{ add\_tag =\> \[ "taskStarted" \]} } if \[trigger\_status\] == "OK" { mutate{ add\_tag =\> \[ "taskTerm… --- ## [Run a filter that checks for values in previous records](https://discuss.elastic.co/t/run-a-filter-that-checks-for-values-in-previous-records/90663)
**Author:** [@Tejas\_Ghutukade](https://discuss.elastic.co/u/Tejas_Ghutukade)\ **Replies:** 3\ **Last updated:** [June 23, 2017, 8:20pm UTC](https://discuss.elastic.co/t/run-a-filter-that-checks-for-values-in-previous-records/90663 "2017-06-23T20:20:39Z")
Hi, i am passing a csv file to lagstach with records of timestamp and processing time. i want to create a fliter that checks if the current timestamp lies within the timestamp and processing time of any of the previous… --- ## [Plugin install via proxy still broken ? ver 5.4.2](https://discuss.elastic.co/t/plugin-install-via-proxy-still-broken-ver-5-4-2/90679)
**Author:** [@Jack\_Judge](https://discuss.elastic.co/u/Jack_Judge)\ **Replies:** 0\ **Last updated:** [June 23, 2017, 7:22pm UTC](https://discuss.elastic.co/t/plugin-install-via-proxy-still-broken-ver-5-4-2/90679 "2017-06-23T19:22:32Z")
Is plugin installation behind proxies still broken in LS 5.4.2 ? This is what I'm getting; (anonymised) export http\_proxy=http://myname:mypass@x.x.x.x:3128 export https\_proxy=http://myname:mypass@x.x.x.x:3128 export '… --- ## [How to run rspec tests in Logstash 5.0.0](https://discuss.elastic.co/t/how-to-run-rspec-tests-in-logstash-5-0-0/64895)
**Author:** [@rcanzanese](https://discuss.elastic.co/u/rcanzanese)\ **Replies:** 2\ **Last updated:** [June 23, 2017, 2:29pm UTC](https://discuss.elastic.co/t/how-to-run-rspec-tests-in-logstash-5-0-0/64895 "2017-06-23T14:29:44Z")
In Logstash 2.4.0, I had ran rpspec tests by installing the development plugins: ./bin/logstash-plugin install --development And then running rspec: ./bin/rspec In Logstash 5.0.0 installing the development plugins … --- ## [Logstash 5.0 Ruby Filter Can't Update Hash in Array](https://discuss.elastic.co/t/logstash-5-0-ruby-filter-cant-update-hash-in-array/90647)
**Author:** [@Java\_Beta](https://discuss.elastic.co/u/Java_Beta)\ **Replies:** 0\ **Last updated:** [June 23, 2017, 2:00pm UTC](https://discuss.elastic.co/t/logstash-5-0-ruby-filter-cant-update-hash-in-array/90647 "2017-06-23T14:00:10Z")
I'm a newbie to Logstash and Ruby, and I meet a subtle problem today. My input JSON like the following: { "1": "1", "2": "2", "market": \[ { "id": "1", "name": "m1" }, { "id": "2", … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=957) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=959)