# One index to rule them all?
**URL:** https://discuss.elastic.co/t/one-index-to-rule-them-all/13957
**Category:** Elasticsearch
**Created:** [October 15, 2013, 8:23pm UTC](https://discuss.elastic.co/t/one-index-to-rule-them-all/13957 "2013-10-15T20:23:03Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@Brian\_O\_Neill](https://discuss.elastic.co/u/Brian_O_Neill)
#### Post date: [October 15, 2013, 8:23pm UTC](https://discuss.elastic.co/t/one-index-to-rule-them-all/13957/1 "2013-10-15T20:23:03Z")
Looking for some advice…
(i'm sure this is a question that has been asked before --- feel free to point me to a url)
We are about to scale a system to accommodate thousands of data feeds. Each data feed contains millions of documents. We need to search across all of those feeds with arbitrarily complex queries. (We will need filtering, custom scoring, etc.)
What are the pros and cons to using a single index for all of those feeds?
Are there any constraints we may run into if we create one index per feed?
Any impact on query time performance/storage requirements, etc?
Or are we safe to implement either given that it all boils down to shards in the end?
-brian
* * *
Brian O'Neill
Chief Architect
Health Market Science
The Science of Better Results
2700 Horizon Drive • King of Prussia, PA • 19406
M: 215.588.6024 • @boneill42 • [healthmarketscience.com](http://healthmarketscience.com)
This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)
#### Post date: [October 15, 2013, 8:52pm UTC](https://discuss.elastic.co/t/one-index-to-rule-them-all/13957/2 "2013-10-15T20:52:24Z")
Have you looked into sharding? basically, that's having the best of both
worlds (if you choose the sharding function wisely!)
I would say go with separate indexes only if you have plans to retire some
of them after time, or do mass-deletes of some sorts, otherwise I don't see
a reason for doing that
On Tue, Oct 15, 2013 at 11:23 PM, Brian O'Neill [bone@alumni.brown.edu](mailto:bone@alumni.brown.edu)wrote:
> Looking for some advice…
> (i'm sure this is a question that has been asked before --- feel free to
> point me to a url)
>
> We are about to scale a system to accommodate thousands of data feeds.
> Each data feed contains millions of documents. We need to search across
> all of those feeds with arbitrarily complex queries. (We will need
> filtering, custom scoring, etc.)
>
> What are the pros and cons to using a single index for all of those feeds?
> Are there any constraints we may run into if we create one index per feed?
>
> Any impact on query time performance/storage requirements, etc?
>
> Or are we safe to implement either given that it all boils down to shards
> in the end?
>
> -brian
>
> * * *
>
> Brian O'Neill
> Chief Architect
> _Health Market Science_
> _The Science of Better Results_
> 2700 Horizon Drive • King of Prussia, PA • 19406\*\*\*\*
> M: 215.588.6024 • @boneill42 [http://www.twitter.com/boneill42](http://www.twitter.com/boneill42) •
> healthmarket [http://healthmarketscience.com/](http://healthmarketscience.com/)science[http://healthmarketscience.com/](http://healthmarketscience.com/)
> .com [http://healthmarketscience.com/](http://healthmarketscience.com/)
>
> This information transmitted in this email message is for the intended
> recipient only and may contain confidential and/or privileged material. If
> you received this email in error and are not the intended recipient, or the
> person responsible to deliver it to the intended recipient, please contact
> the sender at the email above and delete this email and any attachments and
> destroy any copies thereof. Any review, retransmission, dissemination,
> copying or other use of, or taking any action in reliance upon, this
> information by persons or entities other than the intended recipient is
> strictly prohibited.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:12am UTC](https://discuss.elastic.co/t/one-index-to-rule-them-all/13957/3 "2017-07-06T02:12:19Z")