A Delay Between When Data is Archived in Elasticsearch but Visible in Kibana?

Elastic Stack Kibana
1

Hi Elastic Sorcerers,

I’ve recently set up a network monitoring system where data about network activity is archived in an Elasticsearch v 7.4.0 Docker container. On the same physical host (Ubuntu 16.04), I also have a Kibana v 7.4.0 Docker container, which manages and displays the ES data. As I’m new to the Elastic suite of products, everything is pretty much running with minimal (and default) configurations.

The Kibana application works great… except I’ve noticed there is about a three minute lag between when data arrives in Elasticsearch and when that same data appears in my Kibana dashboard.

In other words: Suppose I run a test where Host A sends some traffic to Host B across my network. When I run this test and then do an SQL-like query against Elasticsearch, I can see data about the traffic appear almost immediately. But I must wait three minutes before I see that same data appear in my Kibana Visualizations.

I’ve run a lot of tests, and there is no discrepancy between the ES-reported data and the Kibana-reported data. The only issue is that Kibana seems to be three minutes behind Elasticsearch.
I hope to use my system to monitor network activity in real time, so this three minute lag is a serious problem. But I have no idea where to start troubleshooting.

As I mentioned, I set up my ES / Kibana containers using a tutorial, doing a minimal configuration. I’m sure I’m using a lot of default settings. Is this a Kibana configuration issue, and if so, where would I look?

Or could this be a system problem? When I do a simple “uptime” on the command line of both containers, it seems that they have the same system time.

Any advice or help is appreciated! Thanks….

2

As soon as data is indexed in Elasticsearch, its available for use in Kibana.

Is this a problem with dashboard refresh rate and your dashboard not refreshing fast enough? Or are you seeing problems with queries not returning expected results? Have you checked the time filter on your dashboards? Is it possible that the times are 3 minutes out of sink and the Kibana's now is not the same as Elasticsearch's clock?

3

Hi Nathan, thanks for your questions.

I think a time sync problem is the most likely culprit here. To explain: I've set my Kibana timeframe to be "Last 5 minutes." I've also disabled automatic refresh on the Kibana dashboard.
If I click "Refresh" over and over, it seems like there is a 3 minute gap between the time when I see data records first appear in Elasticsearch and when the same data appears in my Kibana VIsualizations. The rate I use to refresh doesn't have any effect.

Also, I'm not seeing any problems with the data records displayed in Kibana. I've spent a lot of time hand-verifying that if I query ES for a record, and then use a customized Visualization in Kibana to see the exact same data record, the data itself always matches. No issues there.

My only problem is ES can display the data record as soon as the data arrives. But you have to wait three minutes before that same data can be seen in Kibana.

I thought that perhaps my Kibana's clock is three minutes behind my Elasticsearch. That would explain everything, actually. On the command line of both my ES and Kibana Docker containers, a "uptime" Linux command shows that the containers themselves are time-synchronized with the host machine. So I'm guessing that the Kibana app and the Elasticsearch service are not sync'ed properly...? From what I'm describing, would you concur? How could I drill down into the Kibana GUI and check that?

(PS - sorry for the novel)

Many thanks!

4

Click the inspect button to view the the underlying elasticsearch _search request used by Kibana to fetch data.

Just for a sanity check, set the time range to absolute time coving 30 minutes ago to a few days in the future. Do you see any time delays then?

5

Thanks Nathan, will do.

My host machine is currently down for a scheduled OS upgrade, or something. I'm being asked to wait until Monday before I access it again. I'll try your idea as soon as I get the box back.

Thanks!

6

Hi Nathan,

Apologies for the very delayed response. My systems were down for a few days for maintenance, and I was only able to try your suggestions now.

So when I run traffic and then do an inspect on a data record that just appears in Kibana, I see this:

      "range": {
        "@timestamp": {
          "format": "strict_date_optional_time",
          "gte": "2019-11-06T18:45:50.586Z",
          "lte": "2019-11-06T19:00:50.586Z"
        }
      }

But when I do a "uptime" in my Elasticsearch Docker container just one second later, I see this:

[root@f6fafeb52f1b elasticsearch]# uptime
 19:05:02 up 35 days,  5:31,  0 users,  load average: 5.67, 5.99, 4.08
[root@f6fafeb52f1b elasticsearch]#

Its not a perfect test, but it seems that when Kibana is requesting data from 18:45:50 to 19:00:50, Elasticsearch thinks the time is 19:05:02. In other words, Kibana is requesting data from about five minutes in the past, although I have the time frame set to "Last 3 minutes."

I ran a quick back-to-back "uptime" command on both my Elasticsearch and Kibana containers:

ELASTICSEARCH
[root@f6fafeb52f1b elasticsearch]# uptime
 19:30:41 up 35 days,  5:57,  0 users,  load average: 2.13, 2.28, 2.62
[root@f6fafeb52f1b elasticsearch]#

KIBANA
bash-4.2$ uptime
 19:30:40 up 35 days,  5:57,  0 users,  load average: 2.13, 2.28, 2.62
bash-4.2$

As you can see, they are pretty much sync'ed up; that one second delay is really me trying to run both commands as closely together as possible.

It is accurate to say that the Kibana app is somehow 3-5 minutes delayed? That is what my instinct says.

I played around with your suggestions of expanding the timeframe from 30 mins prior into the future... the ~5 minute delay was constant.

Thanks!

7

Hi Nathan,

Sorry to nudge; can we look at this a little more? Or should I open a more focused post?

Thanks,

8

Hi Nathan,

I'm going to repost this question with more specific details. Thank you for your help, I do appreciate it. :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.