I have one log file feeding is about 1-2 mins lagging behind of current time (view from kibana).
The feeding path is like this: logstash shipper -> redis->logstash indexer -> es
and indexer has date filter configured as below, where EventTime is from grok
match => ["EventTime", "HH:mm:ss.SSS YYYY-MM-dd"]
remove_field => ["EventTime"]
I'm not sure which part caused the deplay, so what i did is removed the above date filter from indexer (note i never removed the grok) so that i'll have two time: @timestamp and EventTime. so that i can check whether shipper is the one caused the problem. however, surprisingly, after I made the above change, it's no longer lagging (actually lags about 10s, but it's acceptable). Does this mean date filter is actually quite costly? is it recommended to use?