# How to scale text search with very large indices

**URL:** <https://discuss.elastic.co/t/how-to-scale-text-search-with-very-large-indices/43860>\
**Category:** Elasticsearch\
**Created:** [March 9, 2016, 6:42am UTC](https://discuss.elastic.co/t/how-to-scale-text-search-with-very-large-indices/43860 "2016-03-09T06:42:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jpotts18](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpotts18/32/8365_2.png) [@jpotts18](https://discuss.elastic.co/u/jpotts18)\
**Post date:** [March 9, 2016, 6:42am UTC](https://discuss.elastic.co/t/how-to-scale-text-search-with-very-large-indices/43860/1 "2016-03-09T06:42:07Z")

</div>

I am trying to optimize **text search** over a very large set of documents. As a test dataset, I am using the Enron email dataset ~ 3GB, but an actual dataset may be 100GB+. My structure is basically **GET /enron/email/1**.

Constraints:

- Search response times \< 100 ms

- Reduce duplication of entire dataset as much as possible

- The documents can be assumed to be immutable.

- Text may be searched in a wide range of ways. Here are some examples:

It seems from my reading that I should prefer to have shards over replicas if I don't want to duplicate the data. I'd appreciate any insight you can give me. Thanks!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 9, 2016, 7:26am UTC](https://discuss.elastic.co/t/how-to-scale-text-search-with-very-large-indices/43860/2 "2016-03-09T07:26:15Z")

</div>

I would watch this video from elastic{ON}'16: [https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:10pm UTC](https://discuss.elastic.co/t/how-to-scale-text-search-with-very-large-indices/43860/3 "2017-07-05T23:10:02Z")

</div>