While troubleshooting high cpu usage by Filebeat (80% sustained, even when logs are idle), I noticed it showed a large number of files (current: 8811) in the registrar:
"harvester":{"open_files":4,"running":4}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":214,"batches":4,"total":214},"read":{"bytes":24},"write":{"bytes":16075}},"pipeline":{"clients":1,"events":{"active":0,"published":214,"total":214},"queue":{"acked":214}}},"registrar":{"states":{"current":8811,"update":214},"writes":4}}}}
That was surprising, since these files were quite old and never updated. I did see the open_files was only 4, and that seemed correct to me. To test, I purged all the old files from disk, leaving only the last few days worth, and Filebeat CPU quickly dropped back to 1-5%. I now see this in the logs (current:1392):
"harvester":{"open_files":4,"running":4}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":804,"batches":4,"total":804},"read":{"bytes":24},"write":{"bytes":62509}},"pipeline":{"clients":1,"events":{"active":0,"published":625,"total":625},"queue":{"acked":804}}},"registrar":{"states":{"current":1392,"update":804},"writes":4}}}}
What does current represent here? Since these files were not being updated I would expect them to be removed from the registry, but they seem to be consuming resources.
To keep this number lower, is it advisable to have a small number of large files (rolling daily / weekly) rather than a large number of small files (rolling minutely / hourly)?