Important delay in the appearance of logs in Kibana

Elastic Stack Logstash
1

Hi everyone,
I see a problem about my ELK stack currently in pre-production, and I do not understand where it comes from ...
When I view the logs in Kibana, these are systematically between 25 and 40 minutes behind the T time. for exemple, when I am watching logs in Kibana at 16H, I seen logs happend 30 minutes ago.

I had no time lag during my tries that had not a lot of logs, unlike my server that stores a lot of logs
More (+/- 2 million per week) which makes me think that there is a bottleneck somewhere, especially since my VM on which my ELK runs is slightly undersized relative to the volume of data to be processed.

Nevertheless, I have big doubts about Logstash because I can get logs in live on Kibana when I restart the Logstash service.
However, the offset is recreated little by little up to 40 minutes max, and the most constraining is that all these logs that I did not see in live In Kibana are lost when restarting the LS service!

I saw that there were "queue" solutions like Redis, would the setting up of such a server between LS and elasticsearch could resolve my problem ? Knowing that the goal is to have live logs in Kibana ...

Is someone see what can be my problem ?
Thanks.

2

How are logs being sent to Logstash?

3

Logs are sent from à rsyslog server with udp protocol

4

Given it's UDP it's either a timezone difference, or a delay in rsyslog.

LS won't hold onto things for 30 minutes like this.

5

I thinks about that, but the delay is not always the same, it can be from 20 to 35 minutes, and rsyslog is correctly setting for the timezone. But I m asking if LS use the same timezone than the host machine ?

6

Yes it does.

7

Well, I'm looking for issues in timezone settings, and effectively /etc/timezone give me wrong timezone. I changed it for the good one, now, Rsyslog and ELK server have the same timezone. I rebooting the server after that.
After all services started, logs arrived at time but after few minutes, delay is starting again gradually until about +/- 30 minutes ...

8

Is anyone else got an idea ?
Thanks

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.