Filebeat 5.5.2 Connection reset by peer

Bkt · October 6, 2017, 11:28pm

Hello, I am getting connection reset when using filebeat to send the logs to logstash. When I restart filebeat, it does send the prior events but soon repeats the connection reset by peer event. I am using the filebeat version 5.5.2 and logstash version is 5.4.1 (I have two filebeat instances from different machines which send the data to the same logstash on a third machine, one works fine, the other one keeps running into this issue)

2017-10-06T15:41:13-07:00 INFO Starting prospector of type: log; id: 14053815997108757649
2017-10-06T15:41:13-07:00 INFO Prospector with previous states loaded: 1
2017-10-06T15:41:13-07:00 INFO Starting prospector of type: log; id: 3816360967876514380
2017-10-06T15:41:13-07:00 INFO Prospector with previous states loaded: 12
2017-10-06T15:41:13-07:00 INFO Starting prospector of type: log; id: 13958902202340005319
2017-10-06T15:41:13-07:00 INFO Prospector with previous states loaded: 1
2017-10-06T15:41:13-07:00 INFO Starting prospector of type: log; id: 6158945275295326147
2017-10-06T15:41:13-07:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 4
2017-10-06T15:41:43-07:00 INFO Non-zero metrics in the last 30s: publish.events=15 registrar.states.current=15 registrar.states.update=15 registrar.writes=1
2017-10-06T15:42:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:42:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:43:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:43:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:44:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:44:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:44:43-07:00 INFO Harvester started for file: /scratch/alpha_logs/alpha_output.log
2017-10-06T15:45:13-07:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=372 libbeat.logstash.published_and_acked_events=1 libbeat.publisher.published_events=1 publish.events=2 registrar.states.update=2 registrar.writes=2
2017-10-06T15:45:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:46:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:46:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:47:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:47:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:48:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:48:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:49:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:49:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:49:54-07:00 ERR Failed to publish events caused by: write tcp 10.196.27.243:28765->10.242.132.9:5044: write: connection reset by peer
2017-10-06T15:49:54-07:00 INFO Error publishing events (retrying): write tcp 10.196.27.243:28765->10.242.132.9:5044: write: connection reset by peer
2017-10-06T15:50:13-07:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=358 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=1 libbeat.logstash.published_but_not_acked_events=1 libbeat.publisher.published_events=1 publish.events=1 registrar.states.update=1 registrar.writes=1
2017-10-06T15:50:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:51:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:51:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:52:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:52:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:53:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:53:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:54:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:54:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:54:59-07:00 ERR Failed to publish events caused by: write tcp 10.196.27.243:29354->10.242.132.9:5044: write: connection reset by peer
2017-10-06T15:54:59-07:00 INFO Error publishing events (retrying): write tcp 10.196.27.243:29354->10.242.132.9:5044: write: connection reset by peer
2017-10-06T15:55:13-07:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=358 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=1 libbeat.logstash.published_but_not_acked_events=1 libbeat.publisher.published_events=1 publish.events=1 registrar.states.update=1 registrar.writes=1
2017-10-06T15:55:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:56:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:56:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:57:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:57:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:58:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:58:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:59:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:59:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:59:54-07:00 ERR Failed to publish events caused by: write tcp 10.196.27.243:29933->10.242.132.9:5044: write: connection reset by peer
2017-10-06T15:59:54-07:00 INFO Error publishing events (retrying): write tcp 10.196.27.243:29933->10.242.132.9:5044: write: connection reset by peer
2017-10-06T16:00:13-07:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=363 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=1 libbeat.logstash.published_but_not_acked_events=1 libbeat.publisher.published_events=1 publish.events=1 registrar.states.update=1 registrar.writes=1

warkolm · October 6, 2017, 11:36pm

Is there a firewall or load balancer in use somewhere?

Bkt · October 6, 2017, 11:43pm

nope just two boxes on the same network

steffens · October 8, 2017, 11:10am

Can you check the logstash-input-beats version as well?

It might be logstash closing the connection, due to filebeat being idle for quite some time. It's almost 5 minutes from last ACK until next batch events are published.

You might consider to increase the client_inactivity_timeout to 600 seconds (default is 60 seconds).

Bkt · October 10, 2017, 8:02pm

The beats version is 3.1.15, I have made the change to add the client_inactivity_timeout => 600, will update after watching the logs for some time, thank you Steffen for taking the time out to read this and respond

Bkt · October 10, 2017, 8:21pm

seems the client_inactivity_timeout resolved the issue Thank you @steffens for your valuable inputs

system · November 7, 2017, 8:21pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.