How can I verify all logs have been shipped?

Elastic Stack Beats
1

Hello,

I have configured FileBeat to ship a log file to Elasticsearch directly.
After all harvesters have already automatically stopped with a close_inactive message, I ran a count check in the filebeat indices in ES.

While the number of lines in the log file is 689398 according to wc -l, the number of documents in the index is only 6849.

Does this mean that not all logs have been shipped? What should I do?

Thank you

2

Hi @AvivCohn :slight_smile:

Have you check the contents of the registry file? https://www.elastic.co/guide/en/beats/filebeat/current/migration-registry-file.html

At the same time, it might be helpful if you give us some information like the configuration used, version of Filebeat and the size and format of the logs file.

(Moved to Filebeat category)