# Retrieving over a million records in Elasticsearch

**URL:** https://discuss.elastic.co/t/retrieving-over-a-million-records-in-elasticsearch/41339
**Category:** Elasticsearch
**Created:** [February 10, 2016, 1:22am UTC](https://discuss.elastic.co/t/retrieving-over-a-million-records-in-elasticsearch/41339 "2016-02-10T01:22:28Z")
**Posts on this page:** 1
**Showing post:** 2

<div class="post-metadata">

### Author: ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [February 10, 2016, 2:03am UTC](https://discuss.elastic.co/t/retrieving-over-a-million-records-in-elasticsearch/41339/2 "2016-02-10T02:03:09Z")

</div>

> [@shamak](#):
>
> all are master & data nodes

At this point you are better of making 3 of them master eligible and data nodes and the other 3 just data nodes.

> [@shamak](#):
>
> This indicates, that the obvious bottleneck is in the retrieval of the documents from the gateway node to the corresponding shards

Or the scoring. You should see if it gets faster if you sort by `_doc`.

It could also be fetching the `_id`s.

You should use the hot\_threads API to see what is taking the time.

> [@shamak](#):
>
> I believe filters in Elasticsearch use a bitset to represent the matching documents and aggressively caches these bitsets. It would be really useful if I could get access to the bitset and somehow get the mapping to the document \_ids. Would it be possible to write a plug-in to get this data directly?

The bitsets aren't of \_ids. They are at the Lucene segment level and `_id` is a thing Elasticsearch is inserting on top of that. Depending on your query it may not even use the cache - if it needs scores it won't. If it is super fast without the cache (term query) then it'll skip it as well.

> [@shamak](#):
>
> If there is any information I have forgotten to provide, please let me know.

What do you want to do with the results? Elasticsearch's aggregates were built to do interesting things with portions of the documents after apply arbitrary filters. You might have a similar problem. I mean, maybe its one that can be solved with an aggregation. Or maybe it is one that we just need to better understand.

---

_[View the full topic](https://discuss.elastic.co/t/retrieving-over-a-million-records-in-elasticsearch/41339)._