Filebeat is not sending the entries that it missed when having issue with ulimit

Elastic Stack Beats
1

Filebeat had issue with low open limits which was defaulted to 1024. It his this issue probably due to issue with elasticsearch having problem and connection wasn't established

After fixing the ulimit 'ulimit -n 65536' , i had to restart the filebeat. Once restarted, it was able to establish connection to elasticsearch and send logs.

However it didn't send logs that were missed during the time it had issues? Is filebeat not supposed to send olds logs

2

In order to send logs from the beginning I would suggest to remove the registry file from data directory and restart Filebeat. With this, Filebeat will be started with a clean state.

3

Does that mean that filebeat lost track when hit issues with ulimits?. Deleting the registry means it would end up ingesting data twice for those that are ingested already

4

You are right it will ingest all the data from the beginning. However I'm not sure if you can recover your old states. It depends on how your whole setup failed and if Filebeat marked those logs as sent.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.