# How is performance affected on distribution of data over multiple indices

**URL:** <https://discuss.elastic.co/t/how-is-performance-affected-on-distribution-of-data-over-multiple-indices/5716>\
**Category:** Elasticsearch\
**Created:** [October 28, 2011, 11:27am UTC](https://discuss.elastic.co/t/how-is-performance-affected-on-distribution-of-data-over-multiple-indices/5716 "2011-10-28T11:27:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![narinder\_izap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narinder_izap/32/803_2.png) [@narinder\_izap](https://discuss.elastic.co/u/narinder_izap)\
**Post date:** [October 28, 2011, 11:27am UTC](https://discuss.elastic.co/t/how-is-performance-affected-on-distribution-of-data-over-multiple-indices/5716/1 "2011-10-28T11:27:44Z")

</div>

Hi there,  
We are going to have billions of records in the es, so we  
have plan to distribute the data over multiple indices, I want to know how  
it is going to affect the performance, which of the options will be better  
in terms of performance,

# Single Index with data distributed in multiple types

OR

# Multiple Indices

For example:  
_Multiple Indices:_

Index 1:  
"user" all data related to user.

Index 2:  
"media", all the media related to user, (photos, videos etc), which are  
stored in "INDEX 1", related with some id in those documents.

Index 3:  
"comments", all the comments related to users, and media those are stored  
in above two indices.

_Single Index_

"master\_index"

type: user, media, comments

---

<div class="post-metadata">

**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\
**Post date:** [October 28, 2011, 7:52pm UTC](https://discuss.elastic.co/t/how-is-performance-affected-on-distribution-of-data-over-multiple-indices/5716/2 "2011-10-28T19:52:16Z")

</div>

Narinder Kaur wrote:

> We are going to have billions of records in the es, so we have plan  
> to distribute the data over multiple indices, I want to know how it  
> is going to affect the performance, which of the options will be  
> better in terms of performance,
> 
> # Single Index with data distributed in multiple types
> 
> OR
> 
> # Multiple Indices

I would go for single index with types. That will give you more  
flexibility in relating & retrieving the data.

There's no inherent limit to the size of an index. It's just an  
abstraction on top of one or more shards. You'll want to tune the  
number of shards based on how big your docs are, how many nodes,  
etc.

-Drew

---

<div class="post-metadata">

**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\
**Post date:** [October 31, 2011, 8:40am UTC](https://discuss.elastic.co/t/how-is-performance-affected-on-distribution-of-data-over-multiple-indices/5716/3 "2011-10-31T08:40:38Z")

</div>

It also depends on how many times you'll update the index and how fast  
updated docs should pop up in searches.

If you frequently update (a lot) data and you need them pop up in  
(near) realtime, then you should prefer smaller indices.  
Also it gives you the flexibility (but also more work) to split  
indices not only on type but also on date (e.g. for comments etc).

Peter.

On 28 Okt., 20:52, Drew Raines [d...@raines.me](mailto:d...@raines.me) wrote:

> Narinder Kaur wrote:
> 
> > We are going to have billions of records in the es, so we have plan  
> > to distribute the data over multiple indices, I want to know how it  
> > is going to affect the performance, which of the options will be  
> > better in terms of performance,
> 
> > # Single Index with data distributed in multiple types
> > 
> > OR
> > 
> > # Multiple Indices
> 
> I would go for single index with types. That will give you more  
> flexibility in relating & retrieving the data.
> 
> There's no inherent limit to the size of an index. It's just an  
> abstraction on top of one or more shards. You'll want to tune the  
> number of shards based on how big your docs are, how many nodes,  
> etc.
> 
> -Drew

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:50am UTC](https://discuss.elastic.co/t/how-is-performance-affected-on-distribution-of-data-over-multiple-indices/5716/4 "2017-07-06T03:50:25Z")

</div>