Connection Reset to Logstash

Elastic Stack Beats
1

Trying to send Metricbeat to Logstash. Metricbeat logs don't throw any errors, but Logstash shows the following:

[2023-06-29T15:30:02,110][INFO ][org.logstash.beats.BeatsHandler] [local: 192.168.1.78:5045, remote: 192.168.1.77:55099] Handling exception: java.net.SocketException: Connection reset (caused by: java.net.SocketException: Connection reset)
[2023-06-29T15:30:02,111][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.net.SocketException: Connection reset
	at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
	at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:256) ~[netty-buffer-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.93.Final.jar:4.1.93.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]

My metricbeat.yml is:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
http:
  enabled: true
  port: 5067
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
  index.codec: best_compression
setup.dashboards.enabled: false
setup.kibana:
  host: "https://<redacted fqdn>:443"
  space.id: monitor-test
  username: "<redacted>"
  password: "<redacted>"
output.logstash:
  hosts: ["<redacted fqdn>:5045"]
processors:
  - add_host_metadata: ~
#logging.level: debug
monitoring:
  cluster_uuid: <redacted>
  elasticsearch:
    hosts: ["<redacted fqdn>:9200"]
    username: "<redacted>"
    password: "<redacted>"
    protocol: "https"
    ssl:
      enabled: true
      certificate_authorities:
        - |
          -----BEGIN CERTIFICATE-----
<redacted>
          -----END CERTIFICATE-----
2

You should check this Params are explained here.

On the LS side, only param to set is client_inactivity_timeout. Default value is 60sec

3

The reference documentation is usually my first stop.

  • Both client and server are on the same subnet, no firewall involved.
  • TTL isn't explicitly set so it's using the default value of 0.
  • The connection reset is occurring only on initial start of metricbeat before the agent has fully started, no data has yet been sent when the error occurs.