KIbana App (Within Docker Container) Is Five Minutes Behind System Clock?

Elastic Stack Kibana
1

Hi Kibana Masters,

I recently spun up a Docker container ELK pipeline that ends with Elasticsearch 7.4.0 and Kibana 7.4.0 containers. Everything works great… except there is a lag of about five minutes between when data arrives in ES and then when it is visible in Kibana.

Some observations: Both the ES and Kibana containers are running on the same host, and were spun up at the same time. When I do a synchronized “uptime” on each of them, they practically report the same time:

# Elasticsearch Container
[root@f6fafeb52f1b elasticsearch]# uptime
 21:19:40 up 41 days,  7:46,  0 users,  load average: 2.35, 2.54, 2.69
[root@f6fafeb52f1b elasticsearch]#

# Kibana Container
bash-4.2$ uptime
 21:19:40 up 41 days,  7:46,  0 users,  load average: 2.35, 2.54, 2.69
bash-4.2$

So both containers agree on the current time. That’s good.

When I inject test traffic, I use ES’s elasticsearch-sql-cli-7.4.0.jar JAR file to run SQL-like queries. Test data can be seen in these SQL queries perhaps a second or two after I start the test; so I know ES is getting my data in a quick and timely fashion. That’s great.

But here’s the problem: To see the same exact data in Kibana, I have to wait about five minutes before the data appears. (I’ve set the Kibana timeframe to be “Last 5 minutes” and a refresh rate of 1 second.) All the data is there and matches perfectly with what I see in ES… it just takes Kibana much longer to display it.

When I’m in Kibana, and I use the inspect feature to look at the request Kibana sent for a brand-new data record, the timestamp is:

      "range": {
        "@timestamp": {
          "format": "strict_date_optional_time",
          "gte": "2019-11-12T21:27:29.397Z",
          "lte": "2019-11-12T21:30:29.397Z"
        }
      }

But an “uptime” done on the Kibana container’s command line at the exact same moment shows:

bash-4.2$ uptime
 21:34:05 up 41 days,  8:00,  0 users,  load average: 2.60, 2.44, 2.54
bash-4.2$

So it seems Kibana the container thinks that “now” is 21:34 while Kibana the application thinks “now” is 21:27, which would explain my approximate five minute lag between Kibana and ES. In other words, I suspect Kibana the application is five minutes slow, hence all my data appears in a five minute lag.

So, if this is the case, how can I compensate? Thank you.

2

I think the delay is a sum of multiple factors. If you have a lot of data that comes into Elasticsearch at the same time or if you do a lot of transformations in the ingest pipelines/ or analyzing (when you have lots of big string fields), there will be a delay in there, especially since Kibana usually asks more information than a java client asks for.

3

Thanks Marius,

Really appreciate the thoughts. I don't know how large one of my data records are byte-wise, but I have ~60 fields, and maybe 20 of them are strings. There are, admittedly, millions of data records when my pipeline really gets going. Not sure if that kind of load would tax Kibana to its limits or not

Do you think your theory would explain a nearly five minute delay between ES and Kibana?

I was hoping that there would be a simple setting I could tweak to force Kibana to match its clock as closely as possible with Elasticsearch's. Your response makes me think that this is not a common problem, and probably not a trivial solution. (Ulp.) Am I correct?

Thanks!

4

Kibana doesn't do any processing, it only does requests to ES. And until the data is process by ES, it won't add it to any requests made by Kibana. What I would see as an easy test to see if this is the case is to add more nodes to the ES cluster to see if the response time improves.

5

HmmmmmMMMMMmmmmm...

Interesting, thank you. So I'm not experienced enough yet to add multiple ES nodes into my one-node cluster. But I will research it, though, it is a good idea.

Let me ask you this: When my data is flowing through the pipeline, I notice that it is available in ES with 1 or 2 seconds. That's an excellent response time, I'm really happy with that. I know because I'm using command line tools with watch the my index within ES, plus doing SQL-like queries. There's no delay getting the data into ES.

But... again... my root problem is that the data that is available with seconds in ES is not visible in Kibana for ~5 minutes. My sense is that ES is not a processing bottleneck. Given these observations, do you think it is safe to concur? Or might there be some ES reason why Kibana is delayed before it can see the ES data?

Thank you!

6

For those who might be following this post... I found the issue.

The root problem is that my test machine running the Kibana docker container had a system clock that is 5 minutes fast. When I was interfacing with the container, I was using my laptop. The Kibana app, loaded inside my browser, automatically time-synced with my laptop's clock. The difference between my test host clock and my laptop clock explained the problem I've described above.

So, the workaround solution was to advance my laptop's clock by five minutes. Problem solved.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.