Hi,
I'm processing files that comes from an external server; so once copied, files are static and no new content will be added into the file.
Originally I was reading the files in tail mode but I've experienced some issues processing some lines of the files ( were not ingested into ES.) I have a complex logstash configuration with several translate filters + large dictionaries for enrichment.
I made a change from tail to read mode and I have noticed that:
- Files are processed one at a time (In tail mode, If I have 3 files there were open till files aged at the specified time in close_older)
- Processing is much faster.
Is the performance significant affected then when using tail mode?
Thank you
Regards
Ana