We have been having an issue for a few months where some of our larger elasticsearch clusters will get behind on winlogbeats data showing in kibana.
In our architecture, we forward winlogbeats to kafka then use the kafka input plugin in logstash to read the data, then forward the data to elasticsearch.
One thing I started noticing is that only the data sent to kafka, seems to be behind. We forward syslog directly to logstash and that data is always on time and up to date, even when the data going to kafka first is behind. Both the syslog and kafka winlogbeat data is sent to the same logstash nodes.
Has anyone encountered this before?