I have started using filebeat for log shipping for my gitlab logs, and I have a directory structure that is changed every few minutes, each directory contains a log file which contains data that I want to ship to my elasticsearch and create dashboard from it.
I've configured my "filebeat" yaml file to scan such type of directory
I have managed to collect those logs, now I facing new issues that filebeat is not fast enough to collect logs that are finished fast.
e.g:
I execute the pipeline, XXX.log files were created and finished in 3-5 sec then the files are moved to "artifact directory" which causes filebeat to read X lines from the log file and not all the log content,
any idea how can I grub all the log output before it's moved to "artifact directory"?
Forgive me if I'm misunderstanding your setup but why not just have Filebeat collect the logs from the artifact directory? It sounds like all logs eventually get moved there and would stay there long enough for Filebeat to collect them?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.