# How many indexes is too many indexes?

**URL:** <https://discuss.elastic.co/t/how-many-indexes-is-too-many-indexes/20139>\
**Category:** Elasticsearch\
**Created:** [October 8, 2014, 1:19pm UTC](https://discuss.elastic.co/t/how-many-indexes-is-too-many-indexes/20139 "2014-10-08T13:19:26Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![kmoore\_cce](https://avatars.discourse-cdn.com/v4/letter/k/dfb087/32.png) [@kmoore\_cce](https://discuss.elastic.co/u/kmoore_cce)\
**Post date:** [October 10, 2014, 3:55pm UTC](https://discuss.elastic.co/t/how-many-indexes-is-too-many-indexes/20139/6 "2014-10-10T15:55:34Z")

</div>

Thank you for the feedback guys, it is greatly appreciated.  
I had not thought about file descriptors so that gives me another thing to  
think about.

Our daily volume will be pretty high across all of our users, I don't think  
we have a great estimate, but right now we are at about 50 million  
documents a day and ~30 users.  
Our cluster is in EC2 so we can adjust size and nodes basically whenever we  
need to, so I don't think that is a huge issue assuming we get the index  
layout correct.  
At present we are using one large index which is causing some performance  
issues as you would expect. We also did not get our sharding correct  
originally so now we have really large shards.

We have a requirement to keep 90 days of data per user. With an upward  
bound of users at (indeterminate though) 1000 users. So that would be  
90,000 indexes if we did it by day.  
I guess I am wondering if that is a crazy thing to attempt to do, or if it  
makes more sense to break it up weekly or monthly instead in order to keep  
the index count down.

Our documents are usually pretty small (or what I would consider small) at  
\<= 1K, but we will receive them basically constantly.  
So I guess I am looking for tips on how we can layout and breakup indexes  
to get the best performance benefit as we grow.

Again thank you for the feedback. And appreciate anymore in advance!

Thanks,

On Thursday, October 9, 2014 1:18:07 PM UTC-4, gugod wrote:

> Mark Walkom writes:
> 
> > Did you get better writes?  
> > What sort of storage are you on, did you measure before and after, are  
> > you  
> > reaching I/O limits?
> 
> We pump realtime log data and only measure the the overall processing  
> throughput instead of low level IO throughput (we had the data, but we  
> did not correlate those data with setting change). The disk is just some  
> hard drive but not SSD or some hybrid disk. We did not reach disk or  
> network IO limit before and afterwards. FD limits was the only limit we  
> ran into.
> 
> --  
> Cheers,  
> Kang-min Liu

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6a0f4b11-2061-4972-b475-1f04aaf66bbc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6a0f4b11-2061-4972-b475-1f04aaf66bbc%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

_[View the full topic](https://discuss.elastic.co/t/how-many-indexes-is-too-many-indexes/20139)._