[Filebeat] K8s OOM kill causes file corruption

Elastic Stack Beats
1

Hi,

I have the following issue and think it's a bug but wanted to post here before filing a Github issue:

A filebeat container gets killed/restarted because of memory limits within K8s. After the container restart the following log appears and no events are sent from this filebeat pod anymore:

{"level":"warn","timestamp":"2020-10-20T18:21:56.116Z","caller":"memlog/store.go:130","message":"Incomplete or corrupted log file in /usr/share/filebeat/data/xyz/filebeat. Continue with last known complete and consistent state. Reason: unexpected EOF"}

A pod restart resolves the issue though.

Filebeat version 7.9.2 with Logstash output. For log collection I use the Kubernetes autodiscover feature.

Regards
Bernd

2

Hmmm, as indicated in the log line this is a warning and it shouldn't affect the operation of Filebeat. Would you mind reproducing this issue but with Filebeat logging level set to debug (by setting logging.level: debug in the Filebeat configuration)? And then please post the logs from about 15 seconds before the warning log line through 15 seconds after.

Thanks,

Shaunak