Ignore_older

sd_karthik · January 29, 2016, 10:15pm

HI Folks,

What is exactly ignore_older. I have hourly log files on my server where filebeat is running and streaming logs to logstash.
I have ignore_older set to 30m. I stopped logstash process on destination at 12 10PM today adn started it back at 4 10 PM. In the mean time on the filebeat client, 12, 13 , 14, 15 , 16 Hr log files got created. However filebeats was running on the servers(sources).

After starting logstash at 4:10 I thought 12, 13,14,15 hr log files will be lost since it is older than ignore_older time which is set to 30m. But to my wonder, I see 12,13,14,15,16 log files getting streamed fully after starting the log stash back.

Can someone pls expalin me exactly abt ignore_older and also provide some good documentation.

Thanks,
karthik.

ruflin · February 1, 2016, 8:11am

I think best is to have a look at the discussion here: Logstash File Handler w/ Partially Read Files

In your case ignore_older didn't have an affect, as filebeat was running all the time and was starting the harvester as needed when the files were updated. Then LS was blocking the output so filebeat was waiting to get it available again. ignore_older in filebeat has the same behaviour independent if LS is available or not.