FileBeat6.2.4 ERROR logstash/async.go:235 Failed to publish events caused by: write tcp 192.168.1.2:19616->192.168.1.3:5044: write: connection reset by peer

Elastic Stack Beats
1

Hello.
I'm using the filebeat 6.2.4, but I get this error when I run filebeat, how to solve?

2018-09-06T14:00:09.729+0800	DEBUG	[harvester]	log/log.go:85	End of file reached: /var/log/messages; Backoff now.
2018-09-06T14:00:10.729+0800	DEBUG	[harvester]	log/log.go:85	End of file reached: /var/log/messages; Backoff now.
2018-09-06T14:00:10.730+0800	DEBUG	[transport]	transport/client.go:201	handle error: write tcp 192.168.1.2:19616->192.168.1.3:5044: write: connection reset by peer
2018-09-06T14:00:10.730+0800	DEBUG	[logstash]	logstash/async.go:142	2 events out of 2 events sent to logstash host 192.168.1.3:5044. Continue sending
2018-09-06T14:00:10.730+0800	DEBUG	[logstash]	logstash/async.go:99	close connection
2018-09-06T14:00:10.730+0800	DEBUG	[transport]	transport/client.go:114	closing
2018-09-06T14:00:10.730+0800	ERROR	logstash/async.go:235	Failed to publish events caused by: write tcp 192.168.1.2:19616->192.168.1.3:5044: write: connection reset by peer
2018-09-06T14:00:10.731+0800	DEBUG	[logstash]	logstash/async.go:99	close connection
2018-09-06T14:00:11.731+0800	ERROR	pipeline/output.go:92	Failed to publish events: write tcp 192.168.1.2:19616->192.168.1.3:5044: write: connection reset by peer
2018-09-06T14:00:11.731+0800	DEBUG	[logstash]	logstash/async.go:94	connect
2018-09-06T14:00:11.732+0800	DEBUG	[logstash]	logstash/async.go:142	2 events out of 2 events sent to logstash host 192.168.1.3:5044. Continue sending
2018-09-06T14:00:11.765+0800	DEBUG	[registrar]	registrar/registrar.go:228	Processing 2 events
2018-09-06T14:00:11.766+0800	DEBUG	[registrar]	registrar/registrar.go:200	Registrar state updates processed. Count: 2
2018-09-06T14:00:11.766+0800	DEBUG	[registrar]	registrar/registrar.go:259	Write registry file: /opt/filebeat/data/registry
2018-09-06T14:00:11.766+0800	DEBUG	[registrar]	registrar/registrar.go:284	Registry file updated. 1 states written.
2018-09-06T14:00:12.729+0800	DEBUG	[harvester]	log/log.go:85	End of file reached: /var/log/messages; Backoff now.
2018-09-06T14:00:16.729+0800	DEBUG	[harvester]	log/log.go:85	End of file reached: /var/log/messages; Backoff now.
2018-09-06T14:00:19.684+0800	DEBUG	[prospector]	prospector/prospector.go:124	Run prospector
2018-09-06T14:00:19.684+0800	DEBUG	[prospector]	log/prospector.go:147	Start next scan
2018-09-06T14:00:19.685+0800	DEBUG	[prospector]	log/prospector.go:362	Check file for harvesting: /var/log/messages
2018-09-06T14:00:19.685+0800	DEBUG	[prospector]	log/prospector.go:448	Update existing file for harvesting: /var/log/messages, offset: 134093
2018-09-06T14:00:19.685+0800	DEBUG	[prospector]	log/prospector.go:500	Harvester for file is still running: /var/log/messages
2018-09-06T14:00:19.685+0800	DEBUG	[prospector]	log/prospector.go:168	Prospector states cleaned up. Before: 1, After: 1, Pending: 0
2018-09-06T14:00:24.730+0800	DEBUG	[harvester]	log/log.go:85	End of file reached: /var/log/messages; Backoff now.
2018-09-06T14:00:29.685+0800	DEBUG	[prospector]	prospector/prospector.go:124	Run prospector
2018-09-06T14:00:29.685+0800	DEBUG	[prospector]	log/prospector.go:147	Start next scan
2018-09-06T14:00:29.685+0800	DEBUG	[prospector]	log/prospector.go:362	Check file for harvesting: /var/log/messages
2018-09-06T14:00:29.685+0800	DEBUG	[prospector]	log/prospector.go:448	Update existing file for harvesting: /var/log/messages, offset: 134093
2018-09-06T14:00:29.685+0800	DEBUG	[prospector]	log/prospector.go:500	Harvester for file is still running: /var/log/messages
2018-09-06T14:00:29.685+0800	DEBUG	[prospector]	log/prospector.go:168	Prospector states cleaned up. Before: 1, After: 1, Pending: 0
2018-09-06T14:00:34.730+0800	DEBUG	[harvester]	log/log.go:85	End of file reached: /var/log/messages; Backoff now.

This is the filebeat profile:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/messages

output.logstash:
  hosts: ["192.168.1.3:5044"]

logging.level: debug
logging.selectors: ["*"]

this is my 'ls logstash:

input {
    beats {
        port => 5044
    }
}
output {
    file {
        path => "message-%{+YYYY.MM.dd}/%{host}.log"
    }
}
2

What's in you LS logs?

3

There are no logs in LS

[2018-09-06T13:55:48,922][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/opt/logstash/modules/fb_apache/configuration"}
[2018-09-06T13:55:48,945][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/opt/logstash/modules/netflow/configuration"}
[2018-09-06T13:55:49,411][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-09-06T13:55:49,994][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-09-06T13:55:50,325][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-09-06T13:55:52,072][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>48, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-09-06T13:55:52,540][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-09-06T13:55:52,621][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x14d672ca run>"}
[2018-09-06T13:55:52,673][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-09-06T13:55:52,704][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-09-06T13:56:11,351][INFO ][logstash.outputs.file    ] Opening file {:path=>"/opt/logstash/message-2018.09.06/nfjd-dp-hadoop00-node02.jpushoa.com.log"}
[2018-09-06T13:56:11,356][INFO ][logstash.outputs.file    ] Creating directory {:directory=>"/opt/logstash/message-2018.09.06"}
4

Hi, how to solve this problem, thank you！

5

Could you try updating logstash-input-beats plugin of LS?

bin/logstash-plugin update logstash-input-beats
6

Hi, I have tried to update the logstash-input-beats plugin, but the problem still exists. This error will be reported every time I run it. It seems that the log can be collected normally

7

It's possible LS dropping the connection, because your Beat have been inactive for too long.

Have you tried setting client_activity_timeout to a bigger value in LS?
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-client_inactivity_timeout

8

Thank you very much. The problem was solved. I ran it for more than 20 minutes.
I set this parameter to client_inactivity_timeout

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.