# One large index vs. many smaller indexes

**URL:** <https://discuss.elastic.co/t/one-large-index-vs-many-smaller-indexes/19415>\
**Category:** Elasticsearch\
**Created:** [August 22, 2014, 7:08pm UTC](https://discuss.elastic.co/t/one-large-index-vs-many-smaller-indexes/19415 "2014-08-22T19:08:45Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chris\_Neal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_neal/32/3527_2.png) [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)\
**Post date:** [August 22, 2014, 7:08pm UTC](https://discuss.elastic.co/t/one-large-index-vs-many-smaller-indexes/19415/1 "2014-08-22T19:08:45Z")

</div>

Hi all,

As the subject says, I'm wondering about index size vs. number of indexes.

I'm indexing many application log files, currently with an index by day for  
all logs, which will make a very large index. For just a few applications  
in Development, the index is 55GB a day (across 2 servers). In prod with  
all applications, it will be "much more than that". 1TB a day maybe?

I'm wondering if there is value in splitting the indexes by day and by  
application, which would produce more indexes per day, but they would be  
smaller, vs. value in having a single, mammoth index by day alone.

Is it just a resource question? If I have enough RAM/disk/CPU to support a  
"mammoth" index, then I'm fine? Or are there other reasons to (or to not)  
split up indexes?

Very much appreciate your time.  
Chris

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [August 22, 2014, 10:58pm UTC](https://discuss.elastic.co/t/one-large-index-vs-many-smaller-indexes/19415/2 "2014-08-22T22:58:20Z")

</div>

Hi Chris,

Usually, the problem is not that much in terms of indices but shards, which  
are the physical units of data storage (an index being a logical view over  
several shards).

Something to beware of is that shards typically have some constant overhead  
(disk space, file descriptors, memory usage) that does not depend on the  
amount of data that they store. Although it would be ok to have up to a few  
tens of shards per nodes, you should avoid to have eg. thousands of shards  
per node.

if you plan on always adding a filter for a specific application in your  
search requests, then splitting by application makes sense since this will  
make the filter useless at search time, you will just need to query the  
application-specific index. On the other hand if you don't filter by  
application, then splitting data by yourself into smaller indices would be  
pretty equivalent to storing everything in a single index with a higher  
number of shards.

You might want to check out the following resources that talk about  
capacity planning:

- [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/big-data-search-and-analytics/)
- 

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

On Fri, Aug 22, 2014 at 9:08 PM, Chris Neal [chris.neal@derbysoft.net](mailto:chris.neal@derbysoft.net)  
wrote:

> Hi all,
> 
> As the subject says, I'm wondering about index size vs. number of indexes.
> 
> I'm indexing many application log files, currently with an index by day  
> for all logs, which will make a very large index. For just a few  
> applications in Development, the index is 55GB a day (across 2 servers).  
> In prod with all applications, it will be "much more than that". 1TB a  
> day maybe?
> 
> I'm wondering if there is value in splitting the indexes by day and by  
> application, which would produce more indexes per day, but they would be  
> smaller, vs. value in having a single, mammoth index by day alone.
> 
> Is it just a resource question? If I have enough RAM/disk/CPU to support  
> a "mammoth" index, then I'm fine? Or are there other reasons to (or to  
> not) split up indexes?
> 
> Very much appreciate your time.  
> Chris
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com)  
> [https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Adrien Grand

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Chris\_Neal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_neal/32/3527_2.png) [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)\
**Post date:** [August 24, 2014, 10:07pm UTC](https://discuss.elastic.co/t/one-large-index-vs-many-smaller-indexes/19415/3 "2014-08-24T22:07:39Z")

</div>

Adrien,

Thanks so much for the response. It was very helpful. I will check out  
those links on capacity planning for sure.

One followup question. You mention that tens of shards per node would be  
ok. Are you meaning tens of shards from tens of indexes? Or tens of  
shards for a single index? Right now I have two servers configured with  
the index getting 2 shards (one per server), and 1 replica (per server).

Chris

On Fri, Aug 22, 2014 at 5:58 PM, Adrien Grand \<  
[adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote:

> Hi Chris,
> 
> Usually, the problem is not that much in terms of indices but shards,  
> which are the physical units of data storage (an index being a logical view  
> over several shards).
> 
> Something to beware of is that shards typically have some constant  
> overhead (disk space, file descriptors, memory usage) that does not depend  
> on the amount of data that they store. Although it would be ok to have up  
> to a few tens of shards per nodes, you should avoid to have eg. thousands  
> of shards per node.
> 
> if you plan on always adding a filter for a specific application in your  
> search requests, then splitting by application makes sense since this will  
> make the filter useless at search time, you will just need to query the  
> application-specific index. On the other hand if you don't filter by  
> application, then splitting data by yourself into smaller indices would be  
> pretty equivalent to storing everything in a single index with a higher  
> number of shards.
> 
> You might want to check out the following resources that talk about  
> capacity planning:
> 
> - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/big-data-search-and-analytics/)
> - 
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/capacity-planning.html)
> 
> On Fri, Aug 22, 2014 at 9:08 PM, Chris Neal [chris.neal@derbysoft.net](mailto:chris.neal@derbysoft.net)  
> wrote:
> 
> > Hi all,
> > 
> > As the subject says, I'm wondering about index size vs. number of indexes.
> > 
> > I'm indexing many application log files, currently with an index by day  
> > for all logs, which will make a very large index. For just a few  
> > applications in Development, the index is 55GB a day (across 2 servers).  
> > In prod with all applications, it will be "much more than that". 1TB a  
> > day maybe?
> > 
> > I'm wondering if there is value in splitting the indexes by day and by  
> > application, which would produce more indexes per day, but they would be  
> > smaller, vs. value in having a single, mammoth index by day alone.
> > 
> > Is it just a resource question? If I have enough RAM/disk/CPU to support  
> > a "mammoth" index, then I'm fine? Or are there other reasons to (or to  
> > not) split up indexes?
> > 
> > Very much appreciate your time.  
> > Chris
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Adrien Grand
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com)  
> [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAND3Dph9Z1My%2B2%2BQ-NM-sWNn2vT1qktDi6%2BmR-b9rFN-Xc-\_pw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3Dph9Z1My%2B2%2BQ-NM-sWNn2vT1qktDi6%2BmR-b9rFN-Xc-_pw%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [August 25, 2014, 8:44am UTC](https://discuss.elastic.co/t/one-large-index-vs-many-smaller-indexes/19415/4 "2014-08-25T08:44:41Z")

</div>

I meant tens of shards per node. So if you have N nodes with I indices  
which have S shards and R replicas, that would be (I \* S \* (1 + R)) / N.

One shard per node is optimal but doesn't allows for growth: if you add one  
more node, you cannot spread the indexing work load, that is why it is  
common to have a few shards per node in order to allow elasticsearch to  
spread the load in case you would introduce a new node in your cluster to  
improve your cluster capacity.

On Mon, Aug 25, 2014 at 12:07 AM, Chris Neal [chris.neal@derbysoft.net](mailto:chris.neal@derbysoft.net)  
wrote:

> Adrien,
> 
> Thanks so much for the response. It was very helpful. I will check out  
> those links on capacity planning for sure.
> 
> One followup question. You mention that tens of shards per node would be  
> ok. Are you meaning tens of shards from tens of indexes? Or tens of  
> shards for a single index? Right now I have two servers configured with  
> the index getting 2 shards (one per server), and 1 replica (per server).
> 
> Chris
> 
> On Fri, Aug 22, 2014 at 5:58 PM, Adrien Grand \<  
> [adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote:
> 
> > Hi Chris,
> > 
> > Usually, the problem is not that much in terms of indices but shards,  
> > which are the physical units of data storage (an index being a logical view  
> > over several shards).
> > 
> > Something to beware of is that shards typically have some constant  
> > overhead (disk space, file descriptors, memory usage) that does not depend  
> > on the amount of data that they store. Although it would be ok to have up  
> > to a few tens of shards per nodes, you should avoid to have eg. thousands  
> > of shards per node.
> > 
> > if you plan on always adding a filter for a specific application in your  
> > search requests, then splitting by application makes sense since this will  
> > make the filter useless at search time, you will just need to query the  
> > application-specific index. On the other hand if you don't filter by  
> > application, then splitting data by yourself into smaller indices would be  
> > pretty equivalent to storing everything in a single index with a higher  
> > number of shards.
> > 
> > You might want to check out the following resources that talk about  
> > capacity planning:
> > 
> > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/big-data-search-and-analytics/)
> > - 
> > 
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/capacity-planning.html)
> > 
> > On Fri, Aug 22, 2014 at 9:08 PM, Chris Neal [chris.neal@derbysoft.net](mailto:chris.neal@derbysoft.net)  
> > wrote:
> > 
> > > Hi all,
> > > 
> > > As the subject says, I'm wondering about index size vs. number of  
> > > indexes.
> > > 
> > > I'm indexing many application log files, currently with an index by day  
> > > for all logs, which will make a very large index. For just a few  
> > > applications in Development, the index is 55GB a day (across 2 servers).  
> > > In prod with all applications, it will be "much more than that". 1TB a  
> > > day maybe?
> > > 
> > > I'm wondering if there is value in splitting the indexes by day and by  
> > > application, which would produce more indexes per day, but they would be  
> > > smaller, vs. value in having a single, mammoth index by day alone.
> > > 
> > > Is it just a resource question? If I have enough RAM/disk/CPU to  
> > > support a "mammoth" index, then I'm fine? Or are there other reasons to  
> > > (or to not) split up indexes?
> > > 
> > > Very much appreciate your time.  
> > > Chris
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com)  
> > > [https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > Adrien Grand
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > .
> > 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAND3Dph9Z1My%2B2%2BQ-NM-sWNn2vT1qktDi6%2BmR-b9rFN-Xc-\_pw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3Dph9Z1My%2B2%2BQ-NM-sWNn2vT1qktDi6%2BmR-b9rFN-Xc-_pw%40mail.gmail.com)  
> [https://groups.google.com/d/msgid/elasticsearch/CAND3Dph9Z1My%2B2%2BQ-NM-sWNn2vT1qktDi6%2BmR-b9rFN-Xc-\_pw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAND3Dph9Z1My%2B2%2BQ-NM-sWNn2vT1qktDi6%2BmR-b9rFN-Xc-_pw%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Adrien Grand

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5KGu34xCh6e5PKFm30U8mNAf-0acd7%3DQMAVuriL3msyA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5KGu34xCh6e5PKFm30U8mNAf-0acd7%3DQMAVuriL3msyA%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Chris\_Neal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_neal/32/3527_2.png) [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)\
**Post date:** [August 25, 2014, 2:05pm UTC](https://discuss.elastic.co/t/one-large-index-vs-many-smaller-indexes/19415/5 "2014-08-25T14:05:46Z")

</div>

Thanks Adrien!

Very much appreciate your time and help.

Chris

On Mon, Aug 25, 2014 at 3:44 AM, Adrien Grand \<  
[adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote:

> I meant tens of shards per node. So if you have N nodes with I indices  
> which have S shards and R replicas, that would be (I \* S \* (1 + R)) / N.
> 
> One shard per node is optimal but doesn't allows for growth: if you add  
> one more node, you cannot spread the indexing work load, that is why it is  
> common to have a few shards per node in order to allow elasticsearch to  
> spread the load in case you would introduce a new node in your cluster to  
> improve your cluster capacity.
> 
> On Mon, Aug 25, 2014 at 12:07 AM, Chris Neal [chris.neal@derbysoft.net](mailto:chris.neal@derbysoft.net)  
> wrote:
> 
> > Adrien,
> > 
> > Thanks so much for the response. It was very helpful. I will check out  
> > those links on capacity planning for sure.
> > 
> > One followup question. You mention that tens of shards per node would be  
> > ok. Are you meaning tens of shards from tens of indexes? Or tens of  
> > shards for a single index? Right now I have two servers configured with  
> > the index getting 2 shards (one per server), and 1 replica (per server).
> > 
> > Chris
> > 
> > On Fri, Aug 22, 2014 at 5:58 PM, Adrien Grand \<  
> > [adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote:
> > 
> > > Hi Chris,
> > > 
> > > Usually, the problem is not that much in terms of indices but shards,  
> > > which are the physical units of data storage (an index being a logical view  
> > > over several shards).
> > > 
> > > Something to beware of is that shards typically have some constant  
> > > overhead (disk space, file descriptors, memory usage) that does not depend  
> > > on the amount of data that they store. Although it would be ok to have up  
> > > to a few tens of shards per nodes, you should avoid to have eg. thousands  
> > > of shards per node.
> > > 
> > > if you plan on always adding a filter for a specific application in your  
> > > search requests, then splitting by application makes sense since this will  
> > > make the filter useless at search time, you will just need to query the  
> > > application-specific index. On the other hand if you don't filter by  
> > > application, then splitting data by yourself into smaller indices would be  
> > > pretty equivalent to storing everything in a single index with a higher  
> > > number of shards.
> > > 
> > > You might want to check out the following resources that talk about  
> > > capacity planning:
> > > 
> > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/big-data-search-and-analytics/)
> > > - 
> > > 
> > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/capacity-planning.html)
> > > 
> > > On Fri, Aug 22, 2014 at 9:08 PM, Chris Neal [chris.neal@derbysoft.net](mailto:chris.neal@derbysoft.net)  
> > > wrote:
> > > 
> > > > Hi all,
> > > > 
> > > > As the subject says, I'm wondering about index size vs. number of  
> > > > indexes.
> > > > 
> > > > I'm indexing many application log files, currently with an index by day  
> > > > for all logs, which will make a very large index. For just a few  
> > > > applications in Development, the index is 55GB a day (across 2 servers).  
> > > > In prod with all applications, it will be "much more than that". 1TB a  
> > > > day maybe?
> > > > 
> > > > I'm wondering if there is value in splitting the indexes by day and by  
> > > > application, which would produce more indexes per day, but they would be  
> > > > smaller, vs. value in having a single, mammoth index by day alone.
> > > > 
> > > > Is it just a resource question? If I have enough RAM/disk/CPU to  
> > > > support a "mammoth" index, then I'm fine? Or are there other reasons to  
> > > > (or to not) split up indexes?
> > > > 
> > > > Very much appreciate your time.  
> > > > Chris
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com)  
> > > > [https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > > .  
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > Adrien Grand
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com)  
> > > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > > .
> > > 
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/CAND3Dph9Z1My%2B2%2BQ-NM-sWNn2vT1qktDi6%2BmR-b9rFN-Xc-\_pw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3Dph9Z1My%2B2%2BQ-NM-sWNn2vT1qktDi6%2BmR-b9rFN-Xc-_pw%40mail.gmail.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/CAND3Dph9Z1My%2B2%2BQ-NM-sWNn2vT1qktDi6%2BmR-b9rFN-Xc-\_pw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAND3Dph9Z1My%2B2%2BQ-NM-sWNn2vT1qktDi6%2BmR-b9rFN-Xc-_pw%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> > .
> > 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Adrien Grand
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5KGu34xCh6e5PKFm30U8mNAf-0acd7%3DQMAVuriL3msyA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5KGu34xCh6e5PKFm30U8mNAf-0acd7%3DQMAVuriL3msyA%40mail.gmail.com)  
> [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5KGu34xCh6e5PKFm30U8mNAf-0acd7%3DQMAVuriL3msyA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5KGu34xCh6e5PKFm30U8mNAf-0acd7%3DQMAVuriL3msyA%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAND3DpjrpaV7NB87%3DLxRmAa%2B0RUgQ3oY%3DtuaB9Bc274e9jG9og%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3DpjrpaV7NB87%3DLxRmAa%2B0RUgQ3oY%3DtuaB9Bc274e9jG9og%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:06am UTC](https://discuss.elastic.co/t/one-large-index-vs-many-smaller-indexes/19415/6 "2017-07-06T01:06:41Z")

</div>


