I see a problem about my ELK stack currently in pre-production, and I do not understand where it comes from ...
When I view the logs in Kibana, these are systematically between 25 and 40 minutes behind the T time. for exemple, when I am watching logs in Kibana at 16H, I seen logs happend 30 minutes ago.
I had no time lag during my tries that had not a lot of logs, unlike my server that stores a lot of logs
More (+/- 2 million per week) which makes me think that there is a bottleneck somewhere, especially since my VM on which my ELK runs is slightly undersized relative to the volume of data to be processed.
Nevertheless, I have big doubts about Logstash because I can get logs in live on Kibana when I restart the Logstash service.
However, the offset is recreated little by little up to 40 minutes max, and the most constraining is that all these logs that I did not see in live In Kibana are lost when restarting the LS service!
I saw that there were "queue" solutions like Redis, would the setting up of such a server between LS and elasticsearch could resolve my problem ? Knowing that the goal is to have live logs in Kibana ...
Is someone see what can be my problem ?