How is Kibana accessing Large amounts of data from Elasticsearch in few seconds?

[Harish_fragger]

I am trying to understand how is kibana fetching 129,306,319 hits from Elasticsearch and displaying it in Discover -> Inspect -> Responses?

I can see from the request query that they are retrieving only 500 records at a time but then in the Response section I can see that they are displaying all the hits ie. 129,306,319 hits and on top of that it says that it took only 12489ms to finish the request which is bizarre because I have been trying to do the same using Python and I am not even able to retrieve those records within 20 mins. Please refer to the image for better undersstanding

So how exactly is Kibana processing

1. Such huge volumes of data
2. Is the execution time displayed is for retrieving 500 records or 129,306,319?
3. How are they populating the responses section with all the HITS data in such low time? do they keep storing the data somewhere else in the backend?

[Tomo_M (Tomohiro Mitani)]

Hi,

That's a very good question and I don't know enough about it. Hopefully one of the experts will add to it.

It is the same as the corresponding Elasticsearch query and there is nothing special for kibana.

You can see the query and response in Request and Response tabs. There should be only 500 documents in the Response tab. Hits(total) is counted using track_total_fits option. Accuracy is also commented on in this document.

As I understand it, total hit is computed by something like bitwise operation on the indices (this does not mean index of Elasticsearch but the general meaning of "index"), and cost consuming _source of documents are not retracted.

[warkolm (Mark Walkom)]

That number is how many hits Elasticsearch says there is, but it's not returning all of them, just the count.

[Harish_fragger]

After doing some experimenting with the responses json data I found out that it does not return all the documents rather it just returns 500 docs.