Filebeat not harvesting all files in a folder

Elastic Stack Beats
1

I have a log folder with 744 files, but filebeat has read only 155 files. Is there a setting that controls max number of files to be read.

I am using docker-compose and mapped bind mount volume of local log directory to filebeat docker container. And the container path is configured in the filebeat.yml file.

Please suggest.

2

Can you share your filebeat configuration?

Any errors in filebeat logs?

Is there an ulimit on maximum number of files filebeat or any process is allowed to open?

There is a harvester_limit setting, but I don't think it's set by default.

3

Other things; permissions, file name pattern, age of file (ignore_older).

4

Thanks. I found that files already read are not accessed again.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.