Best practices to import millions of old log files to es

Elastic Stack Beats
1

hi filebeat experts,

I'm going to import a huge amount of old files into ES, and my situation is there are millions of log files. So the main challenge is how to auto removed the log file once it's completely consumed by filebeat.

basically we can use the close_eof, and clean_removed to make sure all the logs files are send to ES smoothly. But how to know which file is fully consumed by filebeat since there's no a option something like remove/rename a file while EOF reached.

and now my way is either append a Event to each file and then read the es data to find the consumed files or reading the filebeat registry file/log file to get the consumed files.

So anybody who has the similar situation, please share your thoughts here.
thanks.

2

Wy not just use cat + stdin to process the files, then delete once it has finished?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.