Latency problems with Discover tab in Kibana

[reswob]

I don't know if this is a Kibana problem or ES problem or a LS problem.

I have one LS, one Kibana and a five cluster ES data nodes (with one being the master) (six if you count the coordinating server on Kibana).

In Kibana, when I go to the Discover tab, I am missing the most recent 3-15 minutes of events at any time.

When I look at the Monitor tab, the LS says it has minimal latency, Events Emitted avg is in the 850 range and Events Received avg is a little higher in the 870 range (I'm dropping several events types which accounts I believe accounts for the difference).

All ES indications are low as well (search rate, indexing rate, search latency, indexing latency). There are no errors in the logs, so I'm wondering where do I look to start troubleshooting?

Is this a problem with Kibana? ES? Or do I just need another LS?

[tsullivan (Tim Sullivan)]

I would start by looking at the system that adds the timestamps to the documents in your index. (Maybe that's being done by Elasticsearch or maybe the timestamps are saved before the document goes to Elasticsearch.) Is that system clock perhaps a few minutes behind the clock of the client system? That would be an easy fix, so hopefully it's something as simple as that.

Another thing to check, if you do a search directly to Elasticsearch for the most recent document, is it the document you would expect to see, or is it an older one? If ES is doesn't have the recent documents until some time goes by, then you'd want to look closely at your pipeline.

@reswob

When I look at the Monitor tab, the LS says it has minimal latency, Events Emitted avg is in the 850 range and Events Received avg is a little higher in the 870 range (I'm dropping several events types which accounts I believe accounts for the difference).

Is Elasticsearch the output for Logstash or do events go through other queuing system(s) first?

[reswob]

I performed a direct search and it shows that there is a gap in the indexing. Indeed the most recent document in ES is over 15 minutes old. So at the moment I'm following tips on this page:
https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html
If none of those work, I'll be looking for other suggestions.

Time is correct on all systems. Although the ES servers and LS are on UTC and Kibana is on EST

BTW, I'm on 5.6.5

[tsullivan (Tim Sullivan)]

This doesn't look like a Kibana or Discover app issue. Discover is correctly finding the documents for the time range.

@reswob

Events Emitted avg is in the 850 range and Events Received avg is a little higher in the 870 range (I'm dropping several events types which accounts I believe accounts for the difference).

These charts are about Event Rates, which is number of events per second. Received is higher than Emitted because some of the events that are received are not emitted, due to filtering.

There's also a latency chart, which is about the time it takes for your pipeline to process events from filter stage to output stage. What kind of numbers are you seeing in the latency chart?

[reswob]

Latency at the time I first posted: avg 5.47 ms with a high of 7.5 ms.
Current latency: 20 ms with no delay in the display on Discover.

Continuing to try to improve indexing speed based on page above.

[tsullivan (Tim Sullivan)]

• Is Elasticsearch the output of your Logstash pipeline or something else?
• Do you perhaps have multiple timestamp fields in the documents? Might want to double-check that you've set the correct one in the Kibana index pattern.