Filebeat ERR Failed to publish events caused by: EOF

[Gude.ravi (Ravi Gude)]

The following is my filebeat yml. The error I am seeing is .
Filebeat - ERR Failed to publish events caused by: EOF
I understand its a connection closing issue by logstash. I do see the data in elasticsearch.
what can be changed on logstash for fixing the above error. I am not using ssl either on logstash or filebeat.
Or should I just ignore the error since I am getting the data in elasticsearch ? I think a blog on this topic would help.
prospectors:
- input_type: log
paths:
- "/prod/app.log"
fields_under_root: true
fields:
app: true
api_name: dtls
- input_type: log
paths:
- "/prod/perf.log"
fields_under_root: true
fields:
perf: true
api_name: dtls
registry_file: .applog-optset
output:
logstash:
hosts: ["logstash.whatever.com:5041"]
timeout: 30
logging:
to_files: true
files:
path: /var/log/filebeat/
name: filebeat_es_logs.log
keepfiles: 7
level: debug

The following is the log
2017-01-08T13:50:56-05:00 DBG output worker: publish 44 events
2017-01-08T13:50:56-05:00 DBG Try to publish 44 events to logstash with window size 62
2017-01-08T13:50:56-05:00 DBG handle error: EOF
2017-01-08T13:50:56-05:00 DBG closing
2017-01-08T13:50:56-05:00 DBG 0 events out of 44 events sent to logstash. Continue sending
2017-01-08T13:50:56-05:00 DBG close connection
2017-01-08T13:50:56-05:00 ERR Failed to publish events caused by: EOF
2017-01-08T13:50:56-05:00 INFO Error publishing events (retrying): EOF
2017-01-08T13:50:56-05:00 DBG close connection
2017-01-08T13:50:56-05:00 DBG send fail
2017-01-08T13:50:56-05:00 DBG End of file reached: /prod/perf.log; Backoff now.
2017-01-08T13:50:56-05:00 DBG End of file reached: /prod/app.log; Backoff now.
2017-01-08T13:50:57-05:00 DBG connect
2017-01-08T13:50:57-05:00 DBG Try to publish 44 events to logstash with window size 31
2017-01-08T13:50:57-05:00 DBG update current window size: 31
2017-01-08T13:50:57-05:00 DBG 31 events out of 44 events sent to logstash. Continue sending
2017-01-08T13:50:57-05:00 DBG Try to publish 13 events to logstash with window size 47
2017-01-08T13:50:57-05:00 DBG 13 events out of 13 events sent to logstash. Continue sending
2017-01-08T13:50:57-05:00 DBG send completed

[maddin2016]

It first try to send async to logstash with window size 62. Then it decrease it to 31 and successfully send to logstash. @ruflin, @andrewkroh is there a way to set this value smaller then 64 or is this value computed?

[Gude.ravi (Ravi Gude)]

Thank you . Is window size configurable ? I thought the window size is dynamic and depends on the traffic between logstash and filebeat.

[tudor (Tudor Golubenco)]

Do you get any errors on the Logstash side? Logstash, at least in some versions, closes the connection if its internal queue blocks for more than 5 seconds. But there should be logs indicating that this is happening.

[Gude.ravi (Ravi Gude)]

Yes on logstash side, I do see errors sometimes like closing connection. but so far I didnt see any issues like documents are being dropped. is window size configurable ?

[steffens (Steffen Siering)]

Which logstash version and which beats input plugin version have you installed?

Only the maximum window size is configurable. Some logstash versions are prone to close connections if a batch is too big or takes to long to process within logstash itself. This can result in duplicate events. Worse, if same batch is resend over and over again as is with logstash failing, one can overload logstash itself (and the final output) with duplicates, without ever making progress (known issue for original logstash-forwarder).

For this reason beats try to adapt the window size up to bulk_max_size (default: 2048).

Newer logstash-input-beats plugin has been rewritten (get most recent version) to send intermediate ACKs as keep-alive signal. This way, logstash has not to drop the connection if it gets congested.

[Gude.ravi (Ravi Gude)]

Thanks Steffen. Version I am using is 5.0.1 ( both ). Please let me know if I should migrate to a newer version.

[steffens (Steffen Siering)]

I guess you didn't update the logstash-input-beats plugin? I'm not sure if logstash 5.1.1 ships with most recent plugin version.

From logstash-input-beats plugin changelog I'd suggest to have at least version 3.1.11 installed. You should be able to upgrade the plugin using the bin/logstash-plugin tool shipped with logstash.