Filebeat failed to publish events

Hi all,
I am using ELK version 7.12.1 to harvest data using filebeat and then send them logstash.
I have two nodes of logstash (10.0.0.1:5044 and 10.0.0.2:5044)

My filebeat output config is as following:

output.logstash:
  # The Logstash hosts
  hosts: ["10.0.0.1:5044" , "10.0.0.2:5044" "]
  loadbalance: true
  worker: 2

When I check Elasticsearch, It seems some events have been lost. To understand what is happening, I change logging level of filebeat to debug. Based on the logs, filebeat harvest the lost logs but following message has been found in the log which cannot connect to logstash:
It is noted that nothing has been found in the logstash log at the issue time.

2021-12-14T10:09:04.631+0330	DEBUG	[harvester]	log/log.go:107	End of file reached: D:\log\23_0.log; Backoff now.
2021-12-14T10:09:05.632+0330	DEBUG	[harvester]	log/log.go:107	End of file reached: D:\log\23_0.log; Backoff now.
2021-12-14T10:09:05.634+0330	DEBUG	[transport]	transport/client.go:205	handle error: write tcp 10.0.0.3:64906->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.639+0330	DEBUG	[transport]	transport/client.go:118	closing
2021-12-14T10:09:05.640+0330	DEBUG	[logstash]	logstash/async.go:172	1 events out of 1 events sent to logstash host 10.0.0.2:5044. Continue sending
2021-12-14T10:09:05.640+0330	DEBUG	[logstash]	logstash/async.go:128	close connection
2021-12-14T10:09:05.640+0330	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-12-14T10:09:05.640+0330	INFO	[publisher]	pipeline/retry.go:223	  done
2021-12-14T10:09:05.641+0330	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: write tcp 10.0.0.3:64906->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.641+0330	DEBUG	[logstash]	logstash/async.go:128	close connection
2021-12-14T10:09:05.642+0330	DEBUG	[transport]	transport/client.go:205	handle error: write tcp 10.0.0.3:64909->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.642+0330	DEBUG	[transport]	transport/client.go:118	closing
2021-12-14T10:09:05.642+0330	DEBUG	[logstash]	logstash/async.go:172	1 events out of 1 events sent to logstash host 10.0.0.2:5044. Continue sending
2021-12-14T10:09:05.642+0330	DEBUG	[logstash]	logstash/async.go:128	close connection
2021-12-14T10:09:05.642+0330	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: write tcp 10.0.0.3:64909->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.642+0330	DEBUG	[logstash]	logstash/async.go:128	close connection
2021-12-14T10:09:05.642+0330	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-12-14T10:09:05.642+0330	INFO	[publisher]	pipeline/retry.go:223	  done
2021-12-14T10:09:05.644+0330	DEBUG	[transport]	transport/client.go:205	handle error: write tcp 10.0.0.3:64907->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.644+0330	DEBUG	[transport]	transport/client.go:118	closing
2021-12-14T10:09:05.644+0330	DEBUG	[logstash]	logstash/async.go:172	1 events out of 1 events sent to logstash host 10.0.0.1:5044. Continue sending
2021-12-14T10:09:05.644+0330	DEBUG	[logstash]	logstash/async.go:128	close connection
2021-12-14T10:09:05.644+0330	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: write tcp 10.0.0.3:64907->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.644+0330	DEBUG	[logstash]	logstash/async.go:128	close connection
2021-12-14T10:09:05.644+0330	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-12-14T10:09:05.644+0330	INFO	[publisher]	pipeline/retry.go:223	  done
2021-12-14T10:09:05.651+0330	DEBUG	[transport]	transport/client.go:205	handle error: write tcp 10.0.0.3:64908->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.651+0330	DEBUG	[transport]	transport/client.go:118	closing
2021-12-14T10:09:05.651+0330	DEBUG	[logstash]	logstash/async.go:172	1 events out of 1 events sent to logstash host 10.0.0.1:5044. Continue sending
2021-12-14T10:09:05.651+0330	DEBUG	[logstash]	logstash/async.go:128	close connection
2021-12-14T10:09:05.651+0330	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: write tcp 10.0.0.3:64908->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.651+0330	DEBUG	[logstash]	logstash/async.go:128	close connection
2021-12-14T10:09:05.651+0330	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-12-14T10:09:05.651+0330	INFO	[publisher]	pipeline/retry.go:223	  done
2021-12-14T10:09:06.716+0330	ERROR	[publisher_pipeline_output]	pipeline/output.go:180	failed to publish events: write tcp 10.0.0.3:64907->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:06.716+0330	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(async(tcp://10.0.0.1:5044))
2021-12-14T10:09:06.716+0330	DEBUG	[logstash]	logstash/async.go:120	connect
2021-12-14T10:09:06.716+0330	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-12-14T10:09:06.716+0330	INFO	[publisher]	pipeline/retry.go:223	  done
2021-12-14T10:09:06.717+0330	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(async(tcp://10.0.0.1:5044)) established
2021-12-14T10:09:06.717+0330	DEBUG	[logstash]	logstash/async.go:172	1 events out of 1 events sent to logstash host 10.0.0.1:5044. Continue sending
2021-12-14T10:09:06.718+0330	DEBUG	[publisher]	memqueue/ackloop.go:160	ackloop: receive ack [2: 0, 1]
2021-12-14T10:09:06.726+0330	DEBUG	[publisher]	memqueue/eventloop.go:535	broker ACK events: count=1, start-seq=6, end-seq=6

2021-12-14T10:09:06.726+0330	DEBUG	[acker]	beater/acker.go:59	stateful ack	{"count": 1}
2021-12-14T10:09:06.726+0330	DEBUG	[publisher]	memqueue/ackloop.go:128	ackloop: return ack to broker loop:1
2021-12-14T10:09:06.726+0330	DEBUG	[publisher]	memqueue/ackloop.go:131	ackloop:  done send ack
2021-12-14T10:09:06.726+0330	DEBUG	[registrar]	registrar/registrar.go:264	Processing 1 events
2021-12-14T10:09:06.726+0330	DEBUG	[registrar]	registrar/registrar.go:231	Registrar state updates processed. Count: 1
2021-12-14T10:09:06.726+0330	DEBUG	[registrar]	registrar/registrar.go:201	Registry file updated. 7 active states.
2021-12-14T10:09:06.821+0330	ERROR	[publisher_pipeline_output]	pipeline/output.go:180	failed to publish events: write tcp 10.0.0.3:64909->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:06.994+0330	ERROR	[publisher_pipeline_output]	pipeline/output.go:180	failed to publish events: write tcp 10.0.0.3:64908->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:07.605+0330	ERROR	[publisher_pipeline_output]	pipeline/output.go:180	failed to publish events: write tcp 10.0.0.3:64906->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:07.633+0330	DEBUG	[harvester]	log/log.go:107	End of file reached: D:\log\23_0.log; Backoff now.
2021-12-14T10:09:11.646+0330	DEBUG	[harvester]	log/log.go:107	End of file reached: D:\log\23_0.log; Backoff now.

these issue happened for some events and the others are ok.

Any advise will be so appreciated

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.