# An error in Logstash with Beats tagged

**URL:** <https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 18, 2018, 10:06am UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506 "2018-04-18T10:06:13Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\
**Post date:** [April 18, 2018, 10:06am UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/1 "2018-04-18T10:06:13Z")

</div>

Im getting an error in logstash but it comes from Beats , i guess looking at the logs.

[[main]\<beats] rejectedExecution - Failed to submit a listener notification task. Event loop shut down?

Can someone please help me to fix this issue?

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [April 18, 2018, 10:51am UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/2 "2018-04-18T10:51:59Z")

</div>

> [@Roshan\_r](#):
>
> rejectedExecution - Failed to submit a listener notification task

Can you provide the full stack trace and also version of logstash and the logstash beats input plugin ( `bin/logstash-plugin list --verbose beats`) ?

---

<div class="post-metadata">

**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\
**Post date:** [April 18, 2018, 12:55pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/3 "2018-04-18T12:55:56Z")

</div>

Hello,  
I managed to fix the issue. it was very funny. When i removed the "Hosts" from the logstash config file, it started working. But i have another issue now. Beats output says this:

2018-04-18T12:54:30.468Z ERROR logstash/async.go:235 Failed to publish events caused by: write tcp 10.85.7.194:29868-\>10.85.7.207:5044: write: connection reset by peer  
2018-04-18T12:54:31.468Z ERROR pipeline/output.go:92 Failed to publish events: write tcp 10.85.7.194:29868-\>10.85.7.207:5044: write: connection reset by peer

logstash version is 6.2.3 and filebeat version is also 6.2.3. Please let me know if you need any more information

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [April 18, 2018, 1:13pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/4 "2018-04-18T13:13:44Z")

</div>

1. can you provide your current input section of the logstash configuration?
2. are there any errors on the logstash side?
3. how many beats are sending to logstash? do you have an idea on the event/second rate?

---

<div class="post-metadata">

**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\
**Post date:** [April 18, 2018, 1:18pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/5 "2018-04-18T13:18:21Z")

</div>

Hi Joao,

1. The input section in logstash:  
input {  
beats {  
port =\> 5044  
type =\> "direct"  
codec =\> plain  
{  
charset =\> "ISO-8859-1"  
}  
}  
}
2. I dont see any errors but warnings in the logs.
3. I dont know any idea about that. But the rate of data is very low. Is there any configuration for that?

Let me know, if you need more information.

Thanks

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [April 18, 2018, 1:20pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/6 "2018-04-18T13:20:27Z")

</div>

what is the output section like? are any events at all reaching the logstash outputs?  
can you show the warnings?

---

<div class="post-metadata">

**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\
**Post date:** [April 18, 2018, 1:25pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/7 "2018-04-18T13:25:45Z")

</div>

Please see the output section.

1. output {  
if "\_grokparsefailure" in [tags] {  
elasticsearch {  
hosts =\> "localhost:9200"  
index =\> "cmdc2-error-%{cmdcLogId}"  
document\_type =\> "error\_logs"  
codec =\> "json"  
}  
} else {  
elasticsearch {  
hosts =\> "localhost:9200"  
index =\> "cmdc2-log-%{+YYYY.MM.dd}"  
document\_type =\> "%{target}\_logs"  
codec =\> "json"  
}  
}  
}

2. Warning logs are like this.  
[WARN] 2018-04-18 13:23:11.647 [Ruby-0-Thread-35@[main]\>worker1: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384] elasticsearch - Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"cmdc2-error-%{cmdcLogId}", :\_type=\>"error\_logs", :\_routing=\>nil}, #LogStash::Event:0x6b58e99c], :response=\>{"index"=\>{"\_index"=\>"cmdc2-error-%{cmdcLogId}", "\_type"=\>"error\_logs", "\_id"=\>nil, "status"=\>400, "error"=\>{"type"=\>"invalid\_index\_name\_exception", "reason"=\>"Invalid index name [cmdc2-error-%{cmdcLogId}], must be lowercase", "index\_uuid"=\>"_na_", "index"=\>"cmdc2-error-%{cmdcLogId}"}}}}  
[WARN] 2018-04-18 13:23:11.641 [Ruby-0-Thread-34@[main]\>worker0: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384] elasticsearch - Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"cmdc2-error-%{cmdcLogId}", :\_type=\>"error\_logs", :\_routing=\>nil}, #LogStash::Event:0x6d71ce87], :response=\>{"index"=\>{"\_index"=\>"cmdc2-error-%{cmdcLogId}", "\_type"=\>"error\_logs", "\_id"=\>nil, "status"=\>400, "error"=\>{"type"=\>"invalid\_index\_name\_exception", "reason"=\>"Invalid index name [cmdc2-error-%{cmdcLogId}], must be lowercase", "index\_uuid"=\>"_na_", "index"=\>"cmdc2-error-%{cmdcLogId}"}}}}  
[WARN] 2018-04-18 13:23:11.648 [Ruby-0-Thread-34@[main]\>worker0: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384] elasticsearch - Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"cmdc2-error-%{cmdcLogId}", :\_type=\>"error\_logs", :\_routing=\>nil}, #LogStash::Event:0x21a3abd7], :response=\>{"index"=\>{"\_index"=\>"cmdc2-error-%{cmdcLogId}", "\_type"=\>"error\_logs", "\_id"=\>nil, "status"=\>400, "error"=\>{"type"=\>"invalid\_index\_name\_exception", "reason"=\>"Invalid index name [cmdc2-error-%{cmdcLogId}], must be lowercase", "index\_uuid"=\>"_na_", "index"=\>"cmdc2-error-%{cmdcLogId}"}}}}  
[WARN] 2018-04-18 13:23:11.648 [Ruby-0-Thread-35@[main]\>worker1: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384] elasticsearch - Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"cmdc2-error-%{cmdcLogId}", :\_type=\>"error\_logs", :\_routing=\>nil}, #LogStash::Event:0x37deac09], :response=\>{"index"=\>{"\_index"=\>"cmdc2-error-%{cmdcLogId}", "\_type"=\>"error\_logs", "\_id"=\>nil, "status"=\>400, "error"=\>{"type"=\>"invalid\_index\_name\_exception", "reason"=\>"Invalid index name [cmdc2-error-%{cmdcLogId}], must be lowercase", "index\_uuid"=\>"_na_", "index"=\>"cmdc2-error-%{cmdcLogId}"}}}}

Also, i cant see the data transmitted from beats to elasticsearch. I meant in elasticsearch logs.

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [April 18, 2018, 1:31pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/8 "2018-04-18T13:31:54Z")

</div>

it seems that events with the \_grokparsefailure don't have the `cmdcLogId` field, so logstash is trying to write to an index that is literally called `"cmdc2-error-%{cmdcLogId}"`.

Because elasticsearch doesn't allow index names with these %{} characters, it's rejecting the data, and logstash is continuously retrying. Because logstash isn't able to move forward with this data, it also stops consuming data from beats and eventually rejects the connections from beats, causing the errors you're seeing in filebeat.

Does that make sense? As long as the events are either sent correctly to one of the outputs or dropped you should start seeing data flow correctly without errors.

---

<div class="post-metadata">

**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\
**Post date:** [April 18, 2018, 1:34pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/9 "2018-04-18T13:34:04Z")

</div>

To be frank, the same set up works with Heka --\> Logstash. This is the first time, i am trying with Filebeat.

---

<div class="post-metadata">

**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\
**Post date:** [April 18, 2018, 1:35pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/10 "2018-04-18T13:35:13Z")

</div>

SO, do you want me to remove that index and try?

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [April 18, 2018, 1:38pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/11 "2018-04-18T13:38:14Z")

</div>

No, just to confirm that the data that reaches logstash and, more specifically, reaches the output section either:  
a) contains a tag `_grokparsefailure` + `cmdcLogId` field: this is the first conditional block  
b) doesn't contain the tag `_grokparsefailure` and has a `target` field: this is for the else block

For debugging purposes you can put a `stdout { codec => rubydebug }` before the `if` (in your output section), to see the event before it is sent to elasticsearch

---

<div class="post-metadata">

**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\
**Post date:** [April 18, 2018, 1:40pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/12 "2018-04-18T13:40:22Z")

</div>

SO the output section looks like this:

output {  
stdout { codec =\> rubydebug }  
if "\_grokparsefailure" in [tags] {  
elasticsearch {  
hosts =\> "localhost:9200"  
index =\> "cmdc2-error-%{cmdcLogId}"  
document\_type =\> "error\_logs"  
codec =\> "json"  
}  
} else {  
elasticsearch {  
hosts =\> "localhost:9200"  
index =\> "cmdc2-log-%{+YYYY.MM.dd}"  
document\_type =\> "%{target}\_logs"  
codec =\> "json"  
}  
}

---

<div class="post-metadata">

**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\
**Post date:** [April 18, 2018, 1:42pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/13 "2018-04-18T13:42:29Z")

</div>

Yes, i can see the lines reaching logstash.  
{  
"@timestamp" =\> 2018-04-18T13:41:34.522Z,  
"cmdcInstanceId" =\> nil,  
"type" =\> "direct",  
"fileId" =\> "0",  
"host" =\> "appin1a",  
"offset" =\> 366882,  
"source" =\> "/var/log/nds/cmdc/cmdc.audit",  
"@version" =\> "1",  
"message" =\> "2018/04/18 13:41:25.127 [TWCAsyncProcessor] [TWC-pool-3-thread-1]: INFO: [98291:105377] TWC request=MercurySortRequest ",  
"instanceId" =\> nil,  
"prospector" =\> {  
"type" =\> "log"  
},  
"beat" =\> {  
"name" =\> "appin1a",  
"hostname" =\> "appin1a",  
"version" =\> "6.2.3"  
},  
"cmdcLogId" =\> nil,  
"tags" =\> [  
[0] "beats\_input\_codec\_plain\_applied",  
[1] "\_grokparsefailure"  
]  
}  
{  
"@timestamp" =\> 2018-04-18T13:41:34.522Z,  
"cmdcInstanceId" =\> nil,  
"type" =\> "direct",  
"fileId" =\> "0",  
"host" =\> "appin1a",  
"offset" =\> 367199,  
"source" =\> "/var/log/nds/cmdc/cmdc.audit",  
"@version" =\> "1",  
"message" =\> "2018/04/18 13:41:25.135 [BaseAsyncApi] [CMDC-pool-2-thread-11]: INFO: [98291] CMDC response status=200 CMDC=9ms TWC=7ms #TWC=1 ",  
"instanceId" =\> nil,  
"prospector" =\> {  
"type" =\> "log"  
},  
"beat" =\> {  
"name" =\> "appin1a",  
"hostname" =\> "appin1a",  
"version" =\> "6.2.3"  
},  
"cmdcLogId" =\> nil,  
"tags" =\> [  
[0] "beats\_input\_codec\_plain\_applied",  
[1] "\_grokparsefailure"  
]  
}  
{  
"@timestamp" =\> 2018-04-18T13:41:34.523Z,  
"cmdcInstanceId" =\> nil,  
"type" =\> "direct",  
"fileId" =\> "0",  
"host" =\> "appin1a",  
"offset" =\> 367833,  
"source" =\> "/var/log/nds/cmdc/cmdc.audit",  
"@version" =\> "1",  
"message" =\> "2018/04/18 13:41:26.482 [TWCAsyncProcessor] [TWC-pool-3-thread-2]: INFO: [98292:105378] TWC request=MercurySortRequest ",  
"instanceId" =\> nil,  
"prospector" =\> {  
"type" =\> "log"  
},  
"beat" =\> {  
"name" =\> "appin1a",  
"hostname" =\> "appin1a",  
"version" =\> "6.2.3"  
},  
"cmdcLogId" =\> nil,  
"tags" =\> [  
[0] "beats\_input\_codec\_plain\_applied",  
[1] "\_grokparsefailure"  
]  
}  
{  
"@timestamp" =\> 2018-04-18T13:41:34.523Z,  
"cmdcInstanceId" =\> nil,  
"type" =\> "direct",  
"fileId" =\> "0",  
"host" =\> "appin1a",  
"offset" =\> 368148,  
"source" =\> "/var/log/nds/cmdc/cmdc.audit",  
"@version" =\> "1",  
"message" =\> "2018/04/18 13:41:26.485 [BaseAsyncApi] [CMDC-pool-2-thread-6]: INFO: [98292] CMDC response status=200 CMDC=3ms TWC=3ms #TWC=1 ",  
"instanceId" =\> nil,  
"prospector" =\> {  
"type" =\> "log"  
},  
"beat" =\> {  
"name" =\> "appin1a",  
"hostname" =\> "appin1a",  
"version" =\> "6.2.3"  
},  
"cmdcLogId" =\> nil,  
"tags" =\> [  
[0] "beats\_input\_codec\_plain\_applied",  
[1] "\_grokparsefailure"  
]  
}  
{  
"@timestamp" =\> 2018-04-18T13:41:35.523Z,  
"cmdcInstanceId" =\> nil,  
"type" =\> "direct",  
"fileId" =\> "0",  
"host" =\> "appin1a",  
"offset" =\> 368785,  
"source" =\> "/var/log/nds/cmdc/cmdc.audit",  
"@version" =\> "1",  
"message" =\> "2018/04/18 13:41:34.838 [TWCAsyncProcessor] [TWC-pool-3-thread-1]: INFO: [98293:105379] TWC request=MercurySortRequest ",  
"instanceId" =\> nil,  
"prospector" =\> {  
"type" =\> "log"  
},  
"beat" =\> {  
"name" =\> "appin1a",  
"hostname" =\> "appin1a",  
"version" =\> "6.2.3"  
},  
"cmdcLogId" =\> nil,  
"tags" =\> [  
[0] "beats\_input\_codec\_plain\_applied",  
[1] "\_grokparsefailure"  
]  
}  
{  
"@timestamp" =\> 2018-04-18T13:41:35.523Z,  
"cmdcInstanceId" =\> nil,  
"type" =\> "direct",  
"fileId" =\> "0",  
"host" =\> "appin1a",  
"offset" =\> 369100,  
"source" =\> "/var/log/nds/cmdc/cmdc.audit",  
"@version" =\> "1",  
"message" =\> "2018/04/18 13:41:34.841 [BaseAsyncApi] [CMDC-pool-2-thread-4]: INFO: [98293] CMDC response status=200 CMDC=4ms TWC=3ms #TWC=1 ",  
"instanceId" =\> nil,  
"prospector" =\> {  
"type" =\> "log"  
},  
"beat" =\> {  
"name" =\> "appin1a",  
"hostname" =\> "appin1a",  
"version" =\> "6.2.3"  
},  
"cmdcLogId" =\> nil,  
"tags" =\> [  
[0] "beats\_input\_codec\_plain\_applied",  
[1] "\_grokparsefailure"  
]  
}

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [April 18, 2018, 1:46pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/14 "2018-04-18T13:46:52Z")

</div>

> [@Roshan\_r](#):
>
> "cmdcLogId" =\> nil,

this seems to be the issue, there's a \_grokparsefaiure tag, so the event is sent to:

```auto
if "_grokparsefailure" in [tags] {
  elasticsearch {
    hosts => "localhost:9200"
    index => "cmdc2-error-%{cmdcLogId}"
    document_type => "error_logs"
    codec => "json"
  }
}

```

but there's no value for `cmdcLogId`, so `index => "cmdc2-error-%{cmdcLogId}"` won't be computed correctly.

---

<div class="post-metadata">

**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\
**Post date:** [April 18, 2018, 1:52pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/15 "2018-04-18T13:52:35Z")

</div>

Thats great. I removed that section from output and no more warnings are displayed. How can i make sure that the logs were processed and sent to elasticsearch

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [April 18, 2018, 1:59pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/16 "2018-04-18T13:59:30Z")

</div>

your if conditional strategy makes sense, but I suggest not having an index name that depends on a field that may not exist (in this case, `cmdcLogId`)

---

<div class="post-metadata">

**Author:** ![Roshan\_r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_r/32/9760_2.png) [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\
**Post date:** [April 18, 2018, 2:02pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/17 "2018-04-18T14:02:21Z")

</div>

Ok. I got you. I remove that index and try again. To make sure that logs were sent from logstash to elasticsearch, should i monitor now at elasticsearch side?

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [April 18, 2018, 2:05pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/18 "2018-04-18T14:05:07Z")

</div>

Yes, but also on the logstash side you can query [the logstash api](https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#pipeline-stats) to find out how many events each output is processing.

Typically you'll want to see that the elasticsearch output in the first "if clause" process no events (which means no errors)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 16, 2018, 2:05pm UTC](https://discuss.elastic.co/t/an-error-in-logstash-with-beats-tagged/128506/19 "2018-05-16T14:05:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.