I have started using filebeat for log shipping for my gitlab logs, and I have a directory structure that is changed every few minutes, each directory contains a log file which contains data that I want to ship to my elasticsearch and create dashboard from it.
I've configured my "filebeat" yaml file to scan such type of directory
- type: log enabled: true paths: - /artifacts/*/*/*/*/*/*/*.log fields: log: jobs-log
it starts scanning the system and when it's finished it does not continue scanning new files creation.
any idea how can I ship all the gitlab logs to elk?