# Failed to publish events caused by: read tcp filebeat:33330-\>logstash:5044: read: connection reset by peer **URL:** \ **Category:** Beats\ **Tags:** docker, filebeat\ **Created:** [October 19, 2020, 9:38pm UTC](https://discuss.elastic.co/t/failed-to-publish-events-caused-by-read-tcp-filebeat-33330-logstash-read-connection-reset-by-peer/252615 "2020-10-19T21:38:54Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ShadwDrgn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shadwdrgn/32/85516_2.png) [@ShadwDrgn](https://discuss.elastic.co/u/ShadwDrgn)\ **Post date:** [October 19, 2020, 9:38pm UTC](https://discuss.elastic.co/t/failed-to-publish-events-caused-by-read-tcp-filebeat-33330-logstash-read-connection-reset-by-peer/252615/1 "2020-10-19T21:38:55Z")
my filebeat constantly (1-3 times a second) can't seem to connect to logstash and do it's thing. I'm on the docker images for 7.9.2 Steps i've already taken: confirmed host is resolving. confirmed I can ping logstash from filebeat confirmed client\_inactivity\_timeout of 3000 doesn't fix it. confirmed i can telnet from filebeat to logstash:5044 tried disabling the elastic output in logstash tried turning up bulk\_max\_size to 4096 tried turning queue.mem events up to 4096 tried restarting both containers many many times confirmed elasticsearch health is green filebeat debug logs: ```auto 2020-10-19T21:32:40.485Z DEBUG [logstash] logstash/async.go:120 connect 2020-10-19T21:32:40.485Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer 2020-10-19T21:32:40.485Z INFO [publisher] pipeline/retry.go:223 done 2020-10-19T21:32:40.486Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(async(tcp://logstash:5044)) established 2020-10-19T21:32:40.543Z DEBUG [logstash] logstash/async.go:172 2048 events out of 2048 events sent to logstash host logstash:5044. Continue sending 2020-10-19T21:32:40.587Z DEBUG [transport] transport/client.go:205 handle error: read tcp 172.21.0.3:54968->172.21.0.7:5044: read: connection reset by peer 2020-10-19T21:32:40.587Z DEBUG [transport] transport/client.go:118 closing 2020-10-19T21:32:40.587Z ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: read tcp 172.21.0.3:54968->172.21.0.7:5044: read: connection reset by peer 2020-10-19T21:32:40.587Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer 2020-10-19T21:32:40.587Z INFO [publisher] pipeline/retry.go:223 done 2020-10-19T21:32:40.645Z DEBUG [logstash] logstash/async.go:172 2048 events out of 2048 events sent to logstash host logstash:5044. Continue sending 2020-10-19T21:32:40.646Z DEBUG [logstash] logstash/async.go:128 close connection 2020-10-19T21:32:40.646Z ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: client is not connected 2020-10-19T21:32:40.646Z DEBUG [logstash] logstash/async.go:128 close connection 2020-10-19T21:32:40.646Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer 2020-10-19T21:32:40.646Z INFO [publisher] pipeline/retry.go:223 done 2020-10-19T21:32:42.022Z ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: client is not connected 2020-10-19T21:32:42.022Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(async(tcp://logstash:5044)) 2020-10-19T21:32:42.022Z DEBUG [logstash] logstash/async.go:120 connect ^ repeats ``` logstash debug logs: ```auto 2020-10-19 21:32:08,496 defaultEventExecutorGroup-4-4 WARN The Logger slowlog.logstash.codecs.plain was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@4ee7f563 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting. 2020-10-19 21:32:09,918 defaultEventExecutorGroup-4-5 WARN The Logger slowlog.logstash.codecs.plain was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@4ee7f563 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting. 2020-10-19 21:32:11,780 defaultEventExecutorGroup-4-6 WARN The Logger slowlog.logstash.codecs.plain was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@4ee7f563 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting. 2020-10-19 21:32:13,873 defaultEventExecutorGroup-4-7 WARN The Logger slowlog.logstash.codecs.plain was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@4ee7f563 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting. 2020-10-19 21:32:15,166 defaultEventExecutorGroup-4-8 WARN The Logger slowlog.logstash.codecs.plain was created with the message factory ^ repeats ``` filebeat.yml: ```auto filebeat.modules: - module: system syslog: enabled: true auth: enabled: true filebeat.inputs: - type: container enabled: true paths: -/var/lib/docker/containers/*/*.log stream: all # can be all, stdout or stderr filebeat.autodiscover: providers: - type: docker # https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html # This URL alos contains instructions on multi-line logs hints.enabled: true #================================ Processors =================================== processors: - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_locale: format: offset - add_host_metadata: netinfo.enabled: true output.logstash: hosts: ["logstash:5044"] bulk_max_size: 4098 #============================== Dashboards ===================================== #setup.dashboards: # enabled: true #============================== Kibana ========================================= #setup.kibana: # host: "${KIBANA_HOST}" #============================== Xpack Monitoring =============================== xpack.monitoring: enabled: true elasticsearch: hosts: ["http://elasticsearch:9200"] logging.level: debug ``` logstash.conf: ```auto input { beats { port => 5044 } } output { elasticsearch { hosts => ["http://elasticsearch:9200"] index => "%{[@metadata][beat]}-%{[@metadata][version]}" } } ``` Any ideas? I'm out of troubleshooting ideas entirely. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [October 20, 2020, 1:11am UTC](https://discuss.elastic.co/t/failed-to-publish-events-caused-by-read-tcp-filebeat-33330-logstash-read-connection-reset-by-peer/252615/2 "2020-10-20T01:11:27Z")
This looks similar to [[Solved] Filebeat -\> Logstash : connection reset by peer](https://discuss.elastic.co/t/solved-filebeat-logstash-connection-reset-by-peer/87012). Have you tried the solution mentioned there? ---
**Author:** ![ShadwDrgn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shadwdrgn/32/85516_2.png) [@ShadwDrgn](https://discuss.elastic.co/u/ShadwDrgn)\ **Post date:** [October 20, 2020, 5:49pm UTC](https://discuss.elastic.co/t/failed-to-publish-events-caused-by-read-tcp-filebeat-33330-logstash-read-connection-reset-by-peer/252615/3 "2020-10-20T17:49:53Z")
yes, I tried setting the timeout to 3000, though it doesn't make sense to me for this to be a timeout issue since the default is 60 seconds and my errors are happening less than a second apart constantly. I will add this to my list of things I tried. ---
**Author:** ![ShadwDrgn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shadwdrgn/32/85516_2.png) [@ShadwDrgn](https://discuss.elastic.co/u/ShadwDrgn)\ **Post date:** [October 21, 2020, 1:17am UTC](https://discuss.elastic.co/t/failed-to-publish-events-caused-by-read-tcp-filebeat-33330-logstash-read-connection-reset-by-peer/252615/4 "2020-10-21T01:17:49Z")
I've discovered new logs that mysteriously weren't in the debug logs new update. I've added another output and all my data is going to that. Also somehow logstash wasn't logging the following: ```auto [2020-10-21T01:05:23,581][WARN][io.netty.channel.DefaultChannelPipeline][main][c2b5a3b810e10242062712ba860a97f313ee95531953316f6998c36268391074] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception., org.jruby.exceptions.RuntimeError: (RuntimeError) Invalid FieldReference: `traefik_http_routers_api_tls_domains[0]_sans`, at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.set(org/logstash/ext/JrubyEventExtLibrary.java:121) ~[logstash-core.jar:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.decoded_event_transform.transform(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/decoded_event_transform.rb:12) ~[?:?], at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1415) ~[jruby-complete-9.2.13.0.jar:?], at org.jruby.java.proxies.MapJavaProxy.each(org/jruby/java/proxies/MapJavaProxy.java:601) ~[jruby-complete-9.2.13.0.jar:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.decoded_event_transform.transform(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/decoded_event_transform.rb:12) ~[?:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.codec_callback_listener.process_event(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/codec_callback_listener.rb:22) ~[?:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.patch.accept(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/patch.rb:10) ~[?:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_plain_minus_3_dot_0_dot_6.lib.logstash.codecs.plain.decode(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.0.6/lib/logstash/codecs/plain.rb:35) ~[?:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.patch.accept(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/patch.rb:9) ~[?:?], at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.delegate.method_missing(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83) ~[?:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.message_listener.onNewMessage(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/message_listener.rb:44) ~[?:?], [2020-10-21T01:05:23,600][INFO][org.logstash.beats.BeatsHandler][main][c2b5a3b810e10242062712ba860a97f313ee95531953316f6998c36268391074] [local: 172.21.0.5:5044, remote: 172.21.0.2:54674] Handling exception: (NoMethodError) undefined method `accept' for nil:NilClass, [2020-10-21T01:05:23,600][WARN][io.netty.channel.DefaultChannelPipeline][main][c2b5a3b810e10242062712ba860a97f313ee95531953316f6998c36268391074] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception., org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `accept' for nil:NilClass, at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.message_listener.onNewMessage(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/message_listener.rb:44) ~[?:?], [2020-10-21T01:05:25,612][INFO][org.logstash.beats.BeatsHandler][main][c2b5a3b810e10242062712ba860a97f313ee95531953316f6998c36268391074] [local: 172.21.0.5:5044, remote: 172.21.0.2:54686] Handling exception: (RuntimeError) Invalid FieldReference: `traefik_http_routers_api_tls_domains[0]_sans`, [2020-10-21T01:05:25,613][WARN][io.netty.channel.DefaultChannelPipeline][main][c2b5a3b810e10242062712ba860a97f313ee95531953316f6998c36268391074] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception., org.jruby.exceptions.RuntimeError: (RuntimeError) Invalid FieldReference: `traefik_http_routers_api_tls_domains[0]_sans`, at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.set(org/logstash/ext/JrubyEventExtLibrary.java:121) ~[logstash-core.jar:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.decoded_event_transform.transform(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/decoded_event_transform.rb:12) ~[?:?], at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1415) ~[jruby-complete-9.2.13.0.jar:?], at org.jruby.java.proxies.MapJavaProxy.each(org/jruby/java/proxies/MapJavaProxy.java:601) ~[jruby-complete-9.2.13.0.jar:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.decoded_event_transform.transform(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/decoded_event_transform.rb:12) ~[?:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.codec_callback_listener.process_event(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/codec_callback_listener.rb:22) ~[?:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.patch.accept(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/patch.rb:10) ~[?:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_plain_minus_3_dot_0_dot_6.lib.logstash.codecs.plain.decode(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.0.6/lib/logstash/codecs/plain.rb:35) ~[?:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.patch.accept(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/patch.rb:9) ~[?:?], at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.delegate.method_missing(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83) ~[?:?], at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.message_listener.onNewMessage(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/message_listener.rb:44) ~[?:?], ``` My logstash.conf is literally just an input and an output to an elastic index with no filters or transforming functions at all. Why is it trying to create this mystical traefik\_http\_routers\_api\_tls\_domains[0]\_sans field? ---
**Author:** ![ShadwDrgn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shadwdrgn/32/85516_2.png) [@ShadwDrgn](https://discuss.elastic.co/u/ShadwDrgn)\ **Post date:** [October 31, 2020, 1:28am UTC](https://discuss.elastic.co/t/failed-to-publish-events-caused-by-read-tcp-filebeat-33330-logstash-read-connection-reset-by-peer/252615/5 "2020-10-31T01:28:05Z")
I think this is because of the square brackets somewhere in key value pairs related to a bug in logstash now. Dunno if it's fixed or being fixed and don't remember the issue# or anything though. sorry guys. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 28, 2020, 3:28am UTC](https://discuss.elastic.co/t/failed-to-publish-events-caused-by-read-tcp-filebeat-33330-logstash-read-connection-reset-by-peer/252615/6 "2020-11-28T03:28:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.