Failed to publish events caused by: read tcp filebeat:33330->logstash:5044: read: connection reset by peer

my filebeat constantly (1-3 times a second) can't seem to connect to logstash and do it's thing. I'm on the docker images for 7.9.2
Steps i've already taken:
confirmed host is resolving.
confirmed I can ping logstash from filebeat
confirmed client_inactivity_timeout of 3000 doesn't fix it.
confirmed i can telnet from filebeat to logstash:5044
tried disabling the elastic output in logstash
tried turning up bulk_max_size to 4096
tried turning queue.mem events up to 4096
tried restarting both containers many many times
confirmed elasticsearch health is green

filebeat debug logs:

2020-10-19T21:32:40.485Z        DEBUG   [logstash]      logstash/async.go:120   connect
2020-10-19T21:32:40.485Z        INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2020-10-19T21:32:40.485Z        INFO    [publisher]     pipeline/retry.go:223     done
2020-10-19T21:32:40.486Z        INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(async(tcp://logstash:5044)) established
2020-10-19T21:32:40.543Z        DEBUG   [logstash]      logstash/async.go:172   2048 events out of 2048 events sent to logstash host logstash:5044. Continue sending
2020-10-19T21:32:40.587Z        DEBUG   [transport]     transport/client.go:205 handle error: read tcp 172.21.0.3:54968->172.21.0.7:5044: read: connection reset by peer
2020-10-19T21:32:40.587Z        DEBUG   [transport]     transport/client.go:118 closing
2020-10-19T21:32:40.587Z        ERROR   [logstash]      logstash/async.go:280   Failed to publish events caused by: read tcp 172.21.0.3:54968->172.21.0.7:5044: read: connection reset by peer
2020-10-19T21:32:40.587Z        INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2020-10-19T21:32:40.587Z        INFO    [publisher]     pipeline/retry.go:223     done
2020-10-19T21:32:40.645Z        DEBUG   [logstash]      logstash/async.go:172   2048 events out of 2048 events sent to logstash host logstash:5044. Continue sending
2020-10-19T21:32:40.646Z        DEBUG   [logstash]      logstash/async.go:128   close connection
2020-10-19T21:32:40.646Z        ERROR   [logstash]      logstash/async.go:280   Failed to publish events caused by: client is not connected
2020-10-19T21:32:40.646Z        DEBUG   [logstash]      logstash/async.go:128   close connection
2020-10-19T21:32:40.646Z        INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2020-10-19T21:32:40.646Z        INFO    [publisher]     pipeline/retry.go:223     done
2020-10-19T21:32:42.022Z        ERROR   [publisher_pipeline_output]     pipeline/output.go:180  failed to publish events: client is not connected
2020-10-19T21:32:42.022Z        INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(async(tcp://logstash:5044))
2020-10-19T21:32:42.022Z        DEBUG   [logstash]      logstash/async.go:120   connect
^ repeats

logstash debug logs:

2020-10-19 21:32:08,496 defaultEventExecutorGroup-4-4 WARN The Logger slowlog.logstash.codecs.plain was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@4ee7f563 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting.
2020-10-19 21:32:09,918 defaultEventExecutorGroup-4-5 WARN The Logger slowlog.logstash.codecs.plain was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@4ee7f563 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting.
2020-10-19 21:32:11,780 defaultEventExecutorGroup-4-6 WARN The Logger slowlog.logstash.codecs.plain was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@4ee7f563 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting.
2020-10-19 21:32:13,873 defaultEventExecutorGroup-4-7 WARN The Logger slowlog.logstash.codecs.plain was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@4ee7f563 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting.
2020-10-19 21:32:15,166 defaultEventExecutorGroup-4-8 WARN The Logger slowlog.logstash.codecs.plain was created with the message factory
^ repeats

filebeat.yml:

filebeat.modules:
- module: system
  syslog:
    enabled: true
  auth:
    enabled: true

filebeat.inputs:
- type: container
  enabled: true
  paths:
    -/var/lib/docker/containers/*/*.log
  stream: all # can be all, stdout or stderr

filebeat.autodiscover:
  providers:
    - type: docker
      # https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html
      # This URL alos contains instructions on multi-line logs
      hints.enabled: true

#================================ Processors ===================================
processors:
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_locale:
    format: offset
- add_host_metadata:
    netinfo.enabled: true

output.logstash:
    hosts: ["logstash:5044"]
    bulk_max_size: 4098


#============================== Dashboards =====================================
#setup.dashboards:
#  enabled: true

#============================== Kibana =========================================
#setup.kibana:
#  host: "${KIBANA_HOST}"

#============================== Xpack Monitoring ===============================
xpack.monitoring:
  enabled: true
  elasticsearch:
      hosts: ["http://elasticsearch:9200"]

logging.level: debug

logstash.conf:

input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}

Any ideas? I'm out of troubleshooting ideas entirely.

This looks similar to [Solved] Filebeat -> Logstash : connection reset by peer. Have you tried the solution mentioned there?

yes, I tried setting the timeout to 3000, though it doesn't make sense to me for this to be a timeout issue since the default is 60 seconds and my errors are happening less than a second apart constantly.
I will add this to my list of things I tried.

I've discovered new logs that mysteriously weren't in the debug logs
new update. I've added another output and all my data is going to that. Also somehow logstash wasn't logging the following:

[2020-10-21T01:05:23,581][WARN ][io.netty.channel.DefaultChannelPipeline][main][c2b5a3b810e10242062712ba860a97f313ee95531953316f6998c36268391074] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.,
org.jruby.exceptions.RuntimeError: (RuntimeError) Invalid FieldReference: `traefik_http_routers_api_tls_domains[0]_sans`,
	at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.set(org/logstash/ext/JrubyEventExtLibrary.java:121) ~[logstash-core.jar:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.decoded_event_transform.transform(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/decoded_event_transform.rb:12) ~[?:?],
	at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1415) ~[jruby-complete-9.2.13.0.jar:?],
	at org.jruby.java.proxies.MapJavaProxy.each(org/jruby/java/proxies/MapJavaProxy.java:601) ~[jruby-complete-9.2.13.0.jar:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.decoded_event_transform.transform(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/decoded_event_transform.rb:12) ~[?:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.codec_callback_listener.process_event(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/codec_callback_listener.rb:22) ~[?:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.patch.accept(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/patch.rb:10) ~[?:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_plain_minus_3_dot_0_dot_6.lib.logstash.codecs.plain.decode(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.0.6/lib/logstash/codecs/plain.rb:35) ~[?:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.patch.accept(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/patch.rb:9) ~[?:?],
	at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.delegate.method_missing(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83) ~[?:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.message_listener.onNewMessage(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/message_listener.rb:44) ~[?:?],
[2020-10-21T01:05:23,600][INFO ][org.logstash.beats.BeatsHandler][main][c2b5a3b810e10242062712ba860a97f313ee95531953316f6998c36268391074] [local: 172.21.0.5:5044, remote: 172.21.0.2:54674] Handling exception: (NoMethodError) undefined method `accept' for nil:NilClass,
[2020-10-21T01:05:23,600][WARN ][io.netty.channel.DefaultChannelPipeline][main][c2b5a3b810e10242062712ba860a97f313ee95531953316f6998c36268391074] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.,
org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `accept' for nil:NilClass,
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.message_listener.onNewMessage(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/message_listener.rb:44) ~[?:?],
[2020-10-21T01:05:25,612][INFO ][org.logstash.beats.BeatsHandler][main][c2b5a3b810e10242062712ba860a97f313ee95531953316f6998c36268391074] [local: 172.21.0.5:5044, remote: 172.21.0.2:54686] Handling exception: (RuntimeError) Invalid FieldReference: `traefik_http_routers_api_tls_domains[0]_sans`,
[2020-10-21T01:05:25,613][WARN ][io.netty.channel.DefaultChannelPipeline][main][c2b5a3b810e10242062712ba860a97f313ee95531953316f6998c36268391074] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.,
org.jruby.exceptions.RuntimeError: (RuntimeError) Invalid FieldReference: `traefik_http_routers_api_tls_domains[0]_sans`,
	at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.set(org/logstash/ext/JrubyEventExtLibrary.java:121) ~[logstash-core.jar:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.decoded_event_transform.transform(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/decoded_event_transform.rb:12) ~[?:?],
	at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1415) ~[jruby-complete-9.2.13.0.jar:?],
	at org.jruby.java.proxies.MapJavaProxy.each(org/jruby/java/proxies/MapJavaProxy.java:601) ~[jruby-complete-9.2.13.0.jar:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.decoded_event_transform.transform(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/decoded_event_transform.rb:12) ~[?:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.codec_callback_listener.process_event(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/codec_callback_listener.rb:22) ~[?:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.patch.accept(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/patch.rb:10) ~[?:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_plain_minus_3_dot_0_dot_6.lib.logstash.codecs.plain.decode(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.0.6/lib/logstash/codecs/plain.rb:35) ~[?:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.patch.accept(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/patch.rb:9) ~[?:?],
	at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.delegate.method_missing(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83) ~[?:?],
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_11_minus_java.lib.logstash.inputs.beats.message_listener.onNewMessage(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.11-java/lib/logstash/inputs/beats/message_listener.rb:44) ~[?:?],

My logstash.conf is literally just an input and an output to an elastic index with no filters or transforming functions at all. Why is it trying to create this mystical traefik_http_routers_api_tls_domains[0]_sans field?

I think this is because of the square brackets somewhere in key value pairs related to a bug in logstash now. Dunno if it's fixed or being fixed and don't remember the issue# or anything though. sorry guys.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.