Hello, I am getting connection reset when using filebeat to send the logs to logstash. When I restart filebeat, it does send the prior events but soon repeats the connection reset by peer event. I am using the filebeat version 5.5.2 and logstash version is 5.4.1 (I have two filebeat instances from different machines which send the data to the same logstash on a third machine, one works fine, the other one keeps running into this issue)
2017-10-06T15:41:13-07:00 INFO Starting prospector of type: log; id: 14053815997108757649
2017-10-06T15:41:13-07:00 INFO Prospector with previous states loaded: 1
2017-10-06T15:41:13-07:00 INFO Starting prospector of type: log; id: 3816360967876514380
2017-10-06T15:41:13-07:00 INFO Prospector with previous states loaded: 12
2017-10-06T15:41:13-07:00 INFO Starting prospector of type: log; id: 13958902202340005319
2017-10-06T15:41:13-07:00 INFO Prospector with previous states loaded: 1
2017-10-06T15:41:13-07:00 INFO Starting prospector of type: log; id: 6158945275295326147
2017-10-06T15:41:13-07:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 4
2017-10-06T15:41:43-07:00 INFO Non-zero metrics in the last 30s: publish.events=15 registrar.states.current=15 registrar.states.update=15 registrar.writes=1
2017-10-06T15:42:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:42:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:43:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:43:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:44:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:44:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:44:43-07:00 INFO Harvester started for file: /scratch/alpha_logs/alpha_output.log
2017-10-06T15:45:13-07:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=372 libbeat.logstash.published_and_acked_events=1 libbeat.publisher.published_events=1 publish.events=2 registrar.states.update=2 registrar.writes=2
2017-10-06T15:45:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:46:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:46:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:47:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:47:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:48:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:48:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:49:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:49:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:49:54-07:00 ERR Failed to publish events caused by: write tcp 10.196.27.243:28765->10.242.132.9:5044: write: connection reset by peer
2017-10-06T15:49:54-07:00 INFO Error publishing events (retrying): write tcp 10.196.27.243:28765->10.242.132.9:5044: write: connection reset by peer
2017-10-06T15:50:13-07:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=358 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=1 libbeat.logstash.published_but_not_acked_events=1 libbeat.publisher.published_events=1 publish.events=1 registrar.states.update=1 registrar.writes=1
2017-10-06T15:50:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:51:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:51:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:52:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:52:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:53:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:53:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:54:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:54:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:54:59-07:00 ERR Failed to publish events caused by: write tcp 10.196.27.243:29354->10.242.132.9:5044: write: connection reset by peer
2017-10-06T15:54:59-07:00 INFO Error publishing events (retrying): write tcp 10.196.27.243:29354->10.242.132.9:5044: write: connection reset by peer
2017-10-06T15:55:13-07:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=358 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=1 libbeat.logstash.published_but_not_acked_events=1 libbeat.publisher.published_events=1 publish.events=1 registrar.states.update=1 registrar.writes=1
2017-10-06T15:55:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:56:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:56:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:57:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:57:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:58:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:58:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:59:13-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:59:43-07:00 INFO No non-zero metrics in the last 30s
2017-10-06T15:59:54-07:00 ERR Failed to publish events caused by: write tcp 10.196.27.243:29933->10.242.132.9:5044: write: connection reset by peer
2017-10-06T15:59:54-07:00 INFO Error publishing events (retrying): write tcp 10.196.27.243:29933->10.242.132.9:5044: write: connection reset by peer
2017-10-06T16:00:13-07:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=363 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=1 libbeat.logstash.published_but_not_acked_events=1 libbeat.publisher.published_events=1 publish.events=1 registrar.states.update=1 registrar.writes=1