# \[\[main\]\<beats\] rejectedExecution - Failed to submit a listener notification task. Event loop shut down?

**URL:** <https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 11, 2018, 10:17am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608 "2018-04-11T10:17:36Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![karto](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@karto](https://discuss.elastic.co/u/karto)\
**Post date:** [April 11, 2018, 10:17am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/1 "2018-04-11T10:17:36Z")

</div>

Hello all,  
I'm new to elastic stack and i'm seeking some help.  
The log shipping worked just fine when i had 2 nodes but the second i'v added a third one i had this error when running the config file

help would be much appreciated.

[https://pastebin.com/PdXz1vS9](https://pastebin.com/PdXz1vS9)

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [April 11, 2018, 10:56am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/2 "2018-04-11T10:56:02Z")

</div>

Hi Karto,

Can you pls explain a bit more and also share the filebeat.yml and logstash.yml config here so that i can check and identify the problem.

Thanks,  
Harsh Bajaj

---

<div class="post-metadata">

**Author:** ![karto](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@karto](https://discuss.elastic.co/u/karto)\
**Post date:** [April 11, 2018, 11:17am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/3 "2018-04-11T11:17:03Z")

</div>

Thank you for relying,

So i have one node containing my elk server (logstash 5.6 kibana 5.6and elasticsearch5.6)  
And i'm trying to ship logs from remote servers where filebeat 5.6 is installed.  
The shipping worked fine when i tested with one node but when i tried to add another node i had that error

Running configuration with file input works.

filebeat.yml  
[https://pastebin.com/nVhcZtu6](https://pastebin.com/nVhcZtu6)

logstash.yml  
[https://pastebin.com/zYiSf5xz](https://pastebin.com/zYiSf5xz)

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [April 11, 2018, 1:31pm UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/4 "2018-04-11T13:31:53Z")

</div>

This look like a bug in the logstash beats input, can you run the following command `bin/logstash-plugin list --verbose beats` from your logstash node and add the output to this issue?

---

<div class="post-metadata">

**Author:** ![karto](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@karto](https://discuss.elastic.co/u/karto)\
**Post date:** [April 11, 2018, 8:10pm UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/5 "2018-04-11T20:10:24Z")

</div>

Here's what i get  
logstash-input-beats (3.1.26)

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [April 12, 2018, 3:25am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/6 "2018-04-12T03:25:32Z")

</div>

Hi karto,

i need logstash pipeline conf file o/p here. please share.  
/etc/logstash/conf.d/

---

<div class="post-metadata">

**Author:** ![karto](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@karto](https://discuss.elastic.co/u/karto)\
**Post date:** [April 12, 2018, 8:16am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/7 "2018-04-12T08:16:35Z")

</div>

Hello Harsh

Yes here's my config  
[https://pastebin.com/ySEcajS5](https://pastebin.com/ySEcajS5)

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [April 12, 2018, 8:56am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/8 "2018-04-12T08:56:23Z")

</div>

Hi @karto,

I chceked your config file and found that you have mentioned "host" parameter in "beats" plugin is :

> host=\>"0.0.0.0"

Please comment this and start filebeat and let me know if there is any error.

Thanks,  
Harsh Bajaj

---

<div class="post-metadata">

**Author:** ![karto](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@karto](https://discuss.elastic.co/u/karto)\
**Post date:** [April 12, 2018, 9:04am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/9 "2018-04-12T09:04:03Z")

</div>

Hello @harshbajaj16,

Unfortunately i still have the same error after commenting that line.

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [April 12, 2018, 9:15am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/10 "2018-04-12T09:15:10Z")

</div>

could you please paste your error here ?

because in your previous file there were many errors:

> WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults  
> Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console  
> [INFO] 2018-04-11 10:05:50.847 [[main]\<beats] Server - Starting server on port: 5044  
> [ERROR] 2018-04-11 10:05:57.122 [[main]\<beats] pipeline - A plugin had an unrecoverable error. Will restart this plugin.  
> Plugin: \<LogStash::Inputs::Beats port=\>5044, host=\>"0.0.0.0", ssl=\>true, ssl\_certificate=\>"/etc/pki/tls/certs/logstash-forwarder.crt", ssl\_key=\>"/etc/pki/tls/private/logstash-forwarder.key", id=\>"8f47c33f5ada3a1471fe3b7dae07f6cb7358e76e-1", enable\_metric=\>true, codec=\>\<LogStash::Codecs::Plain id=\>"plain\_f9eff4c6-afdb-4116-bffd-1084dc08348a", enable\_metric=\>true, charset=\>"UTF-8"\>, ssl\_verify\_mode=\>"none", include\_codec\_tag=\>true, ssl\_handshake\_timeout=\>10000, congestion\_threshold=\>5, target\_field\_for\_codec=\>"message", tls\_min\_version=\>1, tls\_max\_version=\>1.2, cipher\_suites=\>["TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_GCM\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_256\_GCM\_SHA384", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_GCM\_SHA256", "TLS\_ECDHE\_RSA\_WITH\_AES\_128\_GCM\_SHA256", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_CBC\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_256\_CBC\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_128\_CBC\_SHA256", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_CBC\_SHA256"], client\_inactivity\_timeout=\>60, executor\_threads=\>8\>  
> Error: Address already in use  
> [INFO] 2018-04-11 10:05:58.123 [[main]\<beats] Server - Starting server on port: 5044  
> [WARN] 2018-04-11 10:05:58.125 [[main]\<beats] AbstractChannel - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x5606c0e8]

could you please confirm that you are getting same error or diff?

---

<div class="post-metadata">

**Author:** ![karto](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@karto](https://discuss.elastic.co/u/karto)\
**Post date:** [April 12, 2018, 9:37am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/11 "2018-04-12T09:37:58Z")

</div>

Yes It's the same error

Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console  
[INFO] 2018-04-12 09:02:06.562 [[main]\<beats] Server - Starting server on port: 5044  
[ERROR] 2018-04-12 09:02:12.843 [[main]\<beats] pipeline - A plugin had an unrecoverable error. Will restart this plugin.  
Plugin: \<LogStash::Inputs::Beats port=\>5044, ssl=\>true, ssl\_certificate=\>"/etc/pki/tls/certs/logstash-forwarder.crt", ssl\_key=\>"/etc/pki/tls/private/logstash-forwarder.key", id=\>"35f68e6919d41d92df9a1b0e31802b31b181a0c5-1", enable\_metric=\>true, codec=\>\<LogStash::Codecs::Plain id=\>"plain\_032fbd3f-f8a9-4ced-80dc-4ef277ec3522", enable\_metric=\>true, charset=\>"UTF-8"\>, host=\>"0.0.0.0", ssl\_verify\_mode=\>"none", include\_codec\_tag=\>true, ssl\_handshake\_timeout=\>10000, congestion\_threshold=\>5, target\_field\_for\_codec=\>"message", tls\_min\_version=\>1, tls\_max\_version=\>1.2, cipher\_suites=\>["TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_GCM\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_256\_GCM\_SHA384", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_GCM\_SHA256", "TLS\_ECDHE\_RSA\_WITH\_AES\_128\_GCM\_SHA256", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_CBC\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_256\_CBC\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_128\_CBC\_SHA256", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_CBC\_SHA256"], client\_inactivity\_timeout=\>60, executor\_threads=\>8\>  
Error: Address already in use  
[INFO] 2018-04-12 09:02:13.844 [[main]\<beats] Server - Starting server on port: 5044  
[WARN] 2018-04-12 09:02:13.846 [[main]\<beats] AbstractChannel - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x861537bc]  
java.util.concurrent.RejectedExecutionException: event executor terminated  
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:840) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]  
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:342) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]  
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:335) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]  
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:765) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]  
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475) [netty-all-4.1.3.Final.jar:4.1.3.Final]  
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-all-4.1.3.Final.jar:4.1.3.Final]  
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-all-4.1.3.Final.jar:4.1.3.Final]  
at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:85) [netty-all-4.1.3.Final.jar:4.1.3.Final]  
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:330) [netty-all-4.1.3.Final.jar:4.1.3.Final]  
at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:281) [netty-all-4.1.3.Final.jar:4.1.3.Final]  
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:277) [netty-all-4.1.3.Final.jar:4.1.3.Final]  
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:259) [netty-all-4.1.3.Final.jar:4.1.3.Final]  
at org.logstash.beats.Server.listen(Server.java:68) [logstash-input-beats-3.1.26.jar:?]  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0\_161]  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0\_161]  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0\_161]  
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0\_161]  
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:436) [?:?]  
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:300) [?:?]  
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:35) [?:?]  
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134) [?:?]  
at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60) [?:?]  
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105) [?:?]  
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) [?:?]  
at org.jruby.evaluator.ASTInterpreter.INTERPRET\_METHOD(ASTInterpreter.java:74) [?:?]  
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182) [?:?]  
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203) [?:?]  
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168) [?:?]  
at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57) [?:?]  
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105) [?:?]  
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) [?:?]  
at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221) [?:?]  
at org.jruby.ast.RescueNode.interpret(RescueNode.java:116) [?:?]  
at org.jruby.ast.EnsureNode.interpret(EnsureNode.java:96) [?:?]  
at org.jruby.ast.BeginNode.interpret(BeginNode.java:83) [?:?]  
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105) [?:?]  
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) [?:?]

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [April 12, 2018, 9:43am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/12 "2018-04-12T09:43:37Z")

</div>

you r are getting this error while starting logstash or filebeat ?

please confirm.

---

<div class="post-metadata">

**Author:** ![karto](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@karto](https://discuss.elastic.co/u/karto)\
**Post date:** [April 12, 2018, 9:51am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/13 "2018-04-12T09:51:07Z")

</div>

While running the beat config in the server containing logstash.  
logstash doesnt show errors and but when debugging filebeat i get

```
2018/04/12 09:47:47.600105 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 172.31.39.192:5044: getsockopt: connection refused
2018/04/12 09:47:48.601441 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 172.31.39.192:5044: getsockopt: connection refused
2018/04/12 09:47:50.602458 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 172.31.39.192:5044: getsockopt: connection refused
```

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [April 12, 2018, 9:54am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/14 "2018-04-12T09:54:51Z")

</div>

> [@karto](#):
>
> ERR Connecting error publishing events (retrying): dial tcp 172.31.39.192:5044: getsockopt

Hi @karto,

it means your logstash is not working properly. please check your logstash logs.

---

<div class="post-metadata">

**Author:** ![karto](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@karto](https://discuss.elastic.co/u/karto)\
**Post date:** [April 12, 2018, 10:58am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/15 "2018-04-12T10:58:31Z")

</div>

Hello @harshbajaj16

After further digging i'v found this error in logstash log

> [FATAL][logstash.runner] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

I think that i'm missing some configuration  
What i'm trying to do is to centralise logs from multiple 'client machines', for that i'v installed the ELK stack on a server machine and filebeat on each client machine.  
Filebeat configuration is the same for each and every client machine sending data on port 5044  
and i have only one configuration file.  
I want to know if there is more configuration to do in order for logstash to to recieve logs from multiple machines.

Thanks

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [April 12, 2018, 11:30am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/16 "2018-04-12T11:30:50Z")

</div>

> [@karto](#):
>
> [FATAL][logstash.runner] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

This error message is showing you are running two logstash on same server thats ok you can run but data directory should be diff. go to the YML file and change the data directory to define new path in "path.data".

Thanks,  
Harsh Bajaj

---

<div class="post-metadata">

**Author:** ![karto](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@karto](https://discuss.elastic.co/u/karto)\
**Post date:** [April 12, 2018, 11:37am UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/17 "2018-04-12T11:37:42Z")

</div>

Hi @harshbajaj16

It is fixed

Thanks alot.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 10, 2018, 6:19pm UTC](https://discuss.elastic.co/t/main-beats-rejectedexecution-failed-to-submit-a-listener-notification-task-event-loop-shut-down/127608/19 "2018-05-10T18:19:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.