# How is Kibana accessing Large amounts of data from Elasticsearch in few seconds?

**URL:** <https://discuss.elastic.co/t/how-is-kibana-accessing-large-amounts-of-data-from-elasticsearch-in-few-seconds/295039>\
**Category:** Kibana\
**Created:** [January 21, 2022, 9:05am UTC](https://discuss.elastic.co/t/how-is-kibana-accessing-large-amounts-of-data-from-elasticsearch-in-few-seconds/295039 "2022-01-21T09:05:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Harish\_fragger](https://avatars.discourse-cdn.com/v4/letter/h/edb3f5/32.png) [@Harish\_fragger](https://discuss.elastic.co/u/Harish_fragger)\
**Post date:** [January 21, 2022, 9:05am UTC](https://discuss.elastic.co/t/how-is-kibana-accessing-large-amounts-of-data-from-elasticsearch-in-few-seconds/295039/1 "2022-01-21T09:05:08Z")

</div>

I am trying to understand how is kibana fetching 129,306,319 hits from Elasticsearch and displaying it in **Discover -\> Inspect -\> Responses**?

I can see from the request query that they are retrieving only 500 records at a time but then in the Response section I can see that they are displaying all the hits ie. 129,306,319 hits and on top of that it says that it took **only 12489ms** to finish the request which is bizarre because I have been trying to do the same using Python and I am not even able to retrieve those records within 20 mins. _Please refer to the image for better undersstanding_

 ![Opera Snapshot_2022-01-21_022923_wm-stl.mssp.softcell.in](https://us1.discourse-cdn.com/elastic/original/3X/5/5/5571a02fd21117b6708b085bc9dafffb5dbbd12a.png)

So how exactly is Kibana processing

1. Such huge volumes of data
2. Is the execution time displayed is for retrieving 500 records or 129,306,319?
3. How are they populating the **responses section** with all the **HITS** data in such low time? do they keep storing the data somewhere else in the backend?

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [January 21, 2022, 11:42am UTC](https://discuss.elastic.co/t/how-is-kibana-accessing-large-amounts-of-data-from-elasticsearch-in-few-seconds/295039/2 "2022-01-21T11:42:51Z")

</div>

Hi,

That's a very good question and I don't know enough about it. Hopefully one of the experts will add to it.

It is the same as the corresponding Elasticsearch query and there is nothing special for kibana.

You can see the query and response in Request and Response tabs. There should be only 500 documents in the Response tab. Hits(total) is counted using [track\_total\_fits](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-your-data.html#track-total-hits) option. Accuracy is also commented on in this document.

As I understand it, total hit is computed by something like bitwise operation on the indices (this does not mean index of Elasticsearch but the general meaning of "index"), and cost consuming \_source of documents are not retracted.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 23, 2022, 11:44pm UTC](https://discuss.elastic.co/t/how-is-kibana-accessing-large-amounts-of-data-from-elasticsearch-in-few-seconds/295039/3 "2022-01-23T23:44:18Z")

</div>

That number is how many hits Elasticsearch says there is, but it's not returning all of them, just the count.

---

<div class="post-metadata">

**Author:** ![Harish\_fragger](https://avatars.discourse-cdn.com/v4/letter/h/edb3f5/32.png) [@Harish\_fragger](https://discuss.elastic.co/u/Harish_fragger)\
**Post date:** [January 24, 2022, 7:20am UTC](https://discuss.elastic.co/t/how-is-kibana-accessing-large-amounts-of-data-from-elasticsearch-in-few-seconds/295039/4 "2022-01-24T07:20:52Z")

</div>

After doing some experimenting with the responses json data I found out that it does not return all the documents rather it just returns 500 docs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2022, 7:21am UTC](https://discuss.elastic.co/t/how-is-kibana-accessing-large-amounts-of-data-from-elasticsearch-in-few-seconds/295039/5 "2022-02-21T07:21:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.