Connection Reset to Logstash

Trying to send Metricbeat to Logstash. Metricbeat logs don't throw any errors, but Logstash shows the following:

[2023-06-29T15:30:02,110][INFO ][org.logstash.beats.BeatsHandler] [local: 192.168.1.78:5045, remote: 192.168.1.77:55099] Handling exception: java.net.SocketException: Connection reset (caused by: java.net.SocketException: Connection reset)
[2023-06-29T15:30:02,111][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.net.SocketException: Connection reset
	at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
	at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:256) ~[netty-buffer-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.93.Final.jar:4.1.93.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.93.Final.jar:4.1.93.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]

My metricbeat.yml is:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
http:
  enabled: true
  port: 5067
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
  index.codec: best_compression
setup.dashboards.enabled: false
setup.kibana:
  host: "https://<redacted fqdn>:443"
  space.id: monitor-test
  username: "<redacted>"
  password: "<redacted>"
output.logstash:
  hosts: ["<redacted fqdn>:5045"]
processors:
  - add_host_metadata: ~
#logging.level: debug
monitoring:
  cluster_uuid: <redacted>
  elasticsearch:
    hosts: ["<redacted fqdn>:9200"]
    username: "<redacted>"
    password: "<redacted>"
    protocol: "https"
    ssl:
      enabled: true
      certificate_authorities:
        - |
          -----BEGIN CERTIFICATE-----
<redacted>
          -----END CERTIFICATE-----

You should check this Params are explained here.

On the LS side, only param to set is client_inactivity_timeout. Default value is 60sec

The reference documentation is usually my first stop.

  • Both client and server are on the same subnet, no firewall involved.
  • TTL isn't explicitly set so it's using the default value of 0.
  • The connection reset is occurring only on initial start of metricbeat before the agent has fully started, no data has yet been sent when the error occurs.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.