Hi all,
I am using ELK version 7.12.1 to harvest data using filebeat and then send them logstash.
I have two nodes of logstash (10.0.0.1:5044 and 10.0.0.2:5044)
My filebeat output config is as following:
output.logstash:
# The Logstash hosts
hosts: ["10.0.0.1:5044" , "10.0.0.2:5044" "]
loadbalance: true
worker: 2
When I check Elasticsearch, It seems some events have been lost. To understand what is happening, I change logging level of filebeat to debug. Based on the logs, filebeat harvest the lost logs but following message has been found in the log which cannot connect to logstash:
It is noted that nothing has been found in the logstash log at the issue time.
2021-12-14T10:09:04.631+0330 DEBUG [harvester] log/log.go:107 End of file reached: D:\log\23_0.log; Backoff now.
2021-12-14T10:09:05.632+0330 DEBUG [harvester] log/log.go:107 End of file reached: D:\log\23_0.log; Backoff now.
2021-12-14T10:09:05.634+0330 DEBUG [transport] transport/client.go:205 handle error: write tcp 10.0.0.3:64906->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.639+0330 DEBUG [transport] transport/client.go:118 closing
2021-12-14T10:09:05.640+0330 DEBUG [logstash] logstash/async.go:172 1 events out of 1 events sent to logstash host 10.0.0.2:5044. Continue sending
2021-12-14T10:09:05.640+0330 DEBUG [logstash] logstash/async.go:128 close connection
2021-12-14T10:09:05.640+0330 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-12-14T10:09:05.640+0330 INFO [publisher] pipeline/retry.go:223 done
2021-12-14T10:09:05.641+0330 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 10.0.0.3:64906->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.641+0330 DEBUG [logstash] logstash/async.go:128 close connection
2021-12-14T10:09:05.642+0330 DEBUG [transport] transport/client.go:205 handle error: write tcp 10.0.0.3:64909->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.642+0330 DEBUG [transport] transport/client.go:118 closing
2021-12-14T10:09:05.642+0330 DEBUG [logstash] logstash/async.go:172 1 events out of 1 events sent to logstash host 10.0.0.2:5044. Continue sending
2021-12-14T10:09:05.642+0330 DEBUG [logstash] logstash/async.go:128 close connection
2021-12-14T10:09:05.642+0330 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 10.0.0.3:64909->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.642+0330 DEBUG [logstash] logstash/async.go:128 close connection
2021-12-14T10:09:05.642+0330 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-12-14T10:09:05.642+0330 INFO [publisher] pipeline/retry.go:223 done
2021-12-14T10:09:05.644+0330 DEBUG [transport] transport/client.go:205 handle error: write tcp 10.0.0.3:64907->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.644+0330 DEBUG [transport] transport/client.go:118 closing
2021-12-14T10:09:05.644+0330 DEBUG [logstash] logstash/async.go:172 1 events out of 1 events sent to logstash host 10.0.0.1:5044. Continue sending
2021-12-14T10:09:05.644+0330 DEBUG [logstash] logstash/async.go:128 close connection
2021-12-14T10:09:05.644+0330 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 10.0.0.3:64907->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.644+0330 DEBUG [logstash] logstash/async.go:128 close connection
2021-12-14T10:09:05.644+0330 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-12-14T10:09:05.644+0330 INFO [publisher] pipeline/retry.go:223 done
2021-12-14T10:09:05.651+0330 DEBUG [transport] transport/client.go:205 handle error: write tcp 10.0.0.3:64908->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.651+0330 DEBUG [transport] transport/client.go:118 closing
2021-12-14T10:09:05.651+0330 DEBUG [logstash] logstash/async.go:172 1 events out of 1 events sent to logstash host 10.0.0.1:5044. Continue sending
2021-12-14T10:09:05.651+0330 DEBUG [logstash] logstash/async.go:128 close connection
2021-12-14T10:09:05.651+0330 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 10.0.0.3:64908->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:05.651+0330 DEBUG [logstash] logstash/async.go:128 close connection
2021-12-14T10:09:05.651+0330 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-12-14T10:09:05.651+0330 INFO [publisher] pipeline/retry.go:223 done
2021-12-14T10:09:06.716+0330 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 10.0.0.3:64907->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:06.716+0330 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(async(tcp://10.0.0.1:5044))
2021-12-14T10:09:06.716+0330 DEBUG [logstash] logstash/async.go:120 connect
2021-12-14T10:09:06.716+0330 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-12-14T10:09:06.716+0330 INFO [publisher] pipeline/retry.go:223 done
2021-12-14T10:09:06.717+0330 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(async(tcp://10.0.0.1:5044)) established
2021-12-14T10:09:06.717+0330 DEBUG [logstash] logstash/async.go:172 1 events out of 1 events sent to logstash host 10.0.0.1:5044. Continue sending
2021-12-14T10:09:06.718+0330 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [2: 0, 1]
2021-12-14T10:09:06.726+0330 DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=1, start-seq=6, end-seq=6
2021-12-14T10:09:06.726+0330 DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
2021-12-14T10:09:06.726+0330 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:1
2021-12-14T10:09:06.726+0330 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2021-12-14T10:09:06.726+0330 DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
2021-12-14T10:09:06.726+0330 DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
2021-12-14T10:09:06.726+0330 DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 7 active states.
2021-12-14T10:09:06.821+0330 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 10.0.0.3:64909->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:06.994+0330 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 10.0.0.3:64908->10.0.0.1:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:07.605+0330 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 10.0.0.3:64906->10.0.0.2:5044: wsasend: An existing connection was forcibly closed by the remote host.
2021-12-14T10:09:07.633+0330 DEBUG [harvester] log/log.go:107 End of file reached: D:\log\23_0.log; Backoff now.
2021-12-14T10:09:11.646+0330 DEBUG [harvester] log/log.go:107 End of file reached: D:\log\23_0.log; Backoff now.
these issue happened for some events and the others are ok.
Any advise will be so appreciated