Unexpected state reading file(cannot allocate memory)

[msunilreddy (Sunil)]

Hi,
I am getting below error when I start filebeat. It was working fine from past one month. Suddenly my filebeat container exited with OOM killed error. I am using filebeat 6.7.2 version.

2019-11-25T22:05:42.769-0800	ERROR	log/log.go:110	Unexpected state reading from /hostfs/opt/caemm/amq/audit.log; error: read /hostfs/opt/caemm/amq/audit.log: cannot allocate memory
2019-11-25T22:05:42.769-0800	ERROR	log/harvester.go:282	Read line error: read /hostfs/opt/caemm/amq/audit.log: cannot allocate memory; File: /hostfs/opt/caemm/amq/audit.log
2019-11-25T22:05:42.770-0800	ERROR	log/log.go:110	Unexpected state reading from /hostfs/opt/caemm/scheduler/ca-sppevents.log; error: read /hostfs/opt/caemm/scheduler/ca-sppevents.log: cannot allocate memory
2019-11-25T22:05:42.770-0800	ERROR	log/harvester.go:282	Read line error: read /hostfs/opt/caemm/scheduler/ca-sppevents.log: cannot allocate memory; File: /hostfs/opt/caemm/scheduler/ca-sppevents.log
2019-11-25T22:05:42.861-0800	ERROR	log/log.go:110	Unexpected state reading from /hostfs/opt/caemm/readserver/tomeeLogs/localhost.2019-10-21.log; error: read /hostfs/opt/caemm/readserver/tomeeLogs/localhost.2019-10-21.log: cannot allocate memory
2019-11-25T22:05:42.861-0800	ERROR	log/harvester.go:282	Read line error: read /hostfs/opt/caemm/readserver/tomeeLogs/localhost.2019-10-21.log: cannot allocate memory; File: /hostfs/opt/caemm/adminui/tomeeLogs/localhost.2019-10-21.log

I have deployed 4 filebeats and 3 are running fine and one filebeat getting above errors while starting.

Could you any one help me how to resolve this issue.

Thanks,
Sunil

[msunilreddy (Sunil)]

Any idea how to resolve it ?

[kvch (Noémi Ványi)]

Was there any problem on the output side? It is possible that there was backpressure, because ES or LS could not keep up with the load and Filebeat had to keep open many files thus the host run out of memory. Can you see anything in the logs of the output?