# Limit on number of indices? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 9, 2011, 3:06am UTC](https://discuss.elastic.co/t/limit-on-number-of-indices/5825 "2011-11-09T03:06:35Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![searchersteve](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/searchersteve/32/3008_2.png) [@searchersteve](https://discuss.elastic.co/u/searchersteve)\ **Post date:** [November 9, 2011, 3:06am UTC](https://discuss.elastic.co/t/limit-on-number-of-indices/5825/1 "2011-11-09T03:06:35Z")
Is it problematic to create a large number of indices? Because the memory drain on my machine is growing with the number of documents I've placed in ES (I've logged some OutOfMemory errors lately), I am working on a purging strategy that is based on expiration dates. To create maximum flexibility in my purging schedule, I am creating a separate index for each calendar date and document source. Indices are named like "2011-11-08-twitter"; if twitter is scheduled for a weekly purge, then I would delete this index on Nov. 15. I can see this means I'll have about 175 indices at any given time (25 sources \* 7 days). And the number will grow as I add more sources. I considered subdividing the indices into separate types according to source name, but then I'd be forced to purge using the DELETE BY query, which others have advised against. I want to simply delete an entire index. Do I have anything to worry about? ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [November 9, 2011, 9:01am UTC](https://discuss.elastic.co/t/limit-on-number-of-indices/5825/2 "2011-11-09T09:01:00Z")
On Tue, 2011-11-08 at 19:06 -0800, searchersteve wrote: > Is it problematic to create a large number of indices? > > Because the memory drain on my machine is growing with the number of > documents I've placed in ES (I've logged some OutOfMemory errors lately), I > am working on a purging strategy that is based on expiration dates. To > create maximum flexibility in my purging schedule, I am creating a separate > index for each calendar date and document source. Indices are named like > "2011-11-08-twitter"; if twitter is scheduled for a weekly purge, then I > would delete this index on Nov. 15. You care correct: purging by deleting old indices is much more efficient than using delete-by-query, which needs to (1) mark all old docs as deleted (2) expunge the old deleted docs by optimizing to new segments. Each index consists of 1 or more primary shards, and each shard has zero or more replica shards. Each shard is a Lucene instance, and thus there is a memory cost for each shard that you have. So two things I'd suggest: 1. Only have one primary shard for each index, and probably just 1 replica shard. 2. Close indices that you don't actually need access to. clint ---
**Author:** ![searchersteve](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/searchersteve/32/3008_2.png) [@searchersteve](https://discuss.elastic.co/u/searchersteve)\ **Post date:** [November 9, 2011, 9:36am UTC](https://discuss.elastic.co/t/limit-on-number-of-indices/5825/3 "2011-11-09T09:36:01Z")
Thanks, Clint. I feel like I've been using ES for a long time now, but I'm still learning every day. What's the difference between deleting and index and closing it? Does closing mean I can reduce memory consumption for now but maintain the option of re-opening the index later if needed, whereas deleting means losing that option? Steve ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [November 9, 2011, 10:14am UTC](https://discuss.elastic.co/t/limit-on-number-of-indices/5825/4 "2011-11-09T10:14:07Z")
less then 10k indices should be ok (=\> but massive open files!) a lot more should be avoided. why splitting them by source? > Because the memory drain on my machine is growing with the number of documents I've placed in ES but this is not solved via splitting the index or what do you mean here? Peter. On Nov 9, 4:06 am, searchersteve [steve...@gmail.com](mailto:steve...@gmail.com) wrote: > Is it problematic to create a large number of indices? > > Because the memory drain on my machine is growing with the number of > documents I've placed in ES (I've logged some OutOfMemory errors lately), I > am working on a purging strategy that is based on expiration dates. To > create maximum flexibility in my purging schedule, I am creating a separate > index for each calendar date and document source. Indices are named like > "2011-11-08-twitter"; if twitter is scheduled for a weekly purge, then I > would delete this index on Nov. 15. > > I can see this means I'll have about 175 indices at any given time (25 > sources \* 7 days). And the number will grow as I add more sources. I > considered subdividing the indices into separate types according to source > name, but then I'd be forced to purge using the DELETE BY query, which > others have advised against. I want to simply delete an entire index. > > Do I have anything to worry about? > > -- > View this message in context:[http://elasticsearch-users.115913.n3.nabble.com/Limit-on-number-of-in](http://elasticsearch-users.115913.n3.nabble.com/Limit-on-number-of-in)... > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [November 9, 2011, 10:57am UTC](https://discuss.elastic.co/t/limit-on-number-of-indices/5825/5 "2011-11-09T10:57:53Z")
Hi Steve > I feel like I've been using ES for a long time now, but I'm still learning > every day. You're not alone 🙂 > What's the difference between deleting and index and closing it? > Does closing mean I can reduce memory consumption for now but maintain the > option of re-opening the index later if needed, whereas deleting means > losing that option? Closing an index means that ES still knows about the index (which uses up a tiny bit of memory) but it doesn't open it, so it is essentially free. You can just reopen the index when you need to access it. c ---
**Author:** ![searchersteve](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/searchersteve/32/3008_2.png) [@searchersteve](https://discuss.elastic.co/u/searchersteve)\ **Post date:** [November 9, 2011, 7:35pm UTC](https://discuss.elastic.co/t/limit-on-number-of-indices/5825/6 "2011-11-09T19:35:38Z")
Splitting the index is a means to an end. It makes it easier for me to delete old data, keeping the total volume of data limited and reducing the strain on memory. That's the thought, anyway. less then 10k indices should be ok (=\> but massive open files!) a lot more should be avoided. why splitting them by source? > Because the memory drain on my machine is growing with the number of documents I've placed in ES but this is not solved via splitting the index or what do you mean here? Peter. On Nov 9, 4:06 am, searchersteve \ wrote: > Is it problematic to create a large number of indices? > > Because the memory drain on my machine is growing with the number of > documents I've placed in ES (I've logged some OutOfMemory errors lately), I > am working on a purging strategy that is based on expiration dates. To > create maximum flexibility in my purging schedule, I am creating a separate > index for each calendar date and document source. Indices are named like > "2011-11-08-twitter"; if twitter is scheduled for a weekly purge, then I > would delete this index on Nov. 15. > > I can see this means I'll have about 175 indices at any given time (25 > sources \* 7 days). And the number will grow as I add more sources. I > considered subdividing the indices into separate types according to source > name, but then I'd be forced to purge using the DELETE BY query, which > others have advised against. I want to simply delete an entire index. > > Do I have anything to worry about? > > -- > View this message in context:[http://elasticsearch-users.115913.n3.nabble.com/Limit-on-number-of-in](http://elasticsearch-users.115913.n3.nabble.com/Limit-on-number-of-in)... > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![searchersteve](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/searchersteve/32/3008_2.png) [@searchersteve](https://discuss.elastic.co/u/searchersteve)\ **Post date:** [November 12, 2011, 12:59am UTC](https://discuss.elastic.co/t/limit-on-number-of-indices/5825/7 "2011-11-12T00:59:10Z")
Just one more question... You suggest, "Only have one primary shard for each index, and probably just 1 replica shard." How badly will this reduce performance when multiple users are hitting the search engine at once? ---
**Author:** ![searchersteve](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/searchersteve/32/3008_2.png) [@searchersteve](https://discuss.elastic.co/u/searchersteve)\ **Post date:** [November 12, 2011, 1:57am UTC](https://discuss.elastic.co/t/limit-on-number-of-indices/5825/8 "2011-11-12T01:57:37Z")
And one more last, last question........ Now I'm reading about the \_TTL (time to live) field and how ES automatically purges old docs. That seems like an even more straightforward approach than creating a separate index for each day's worth of data. I could put all my docs in one index that way. Is THIS the optimal purge solution? I know this treads on ground previously discussed on the list, but there seem to be varying themes over time. Steve ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 13, 2011, 8:44am UTC](https://discuss.elastic.co/t/limit-on-number-of-indices/5825/9 "2011-11-13T08:44:46Z")
TTL still means that those docs are deleted from the index (similar to delete by query), which means they are marked as deleted, and then will eventually need to be merged out of it. It might be ok for your case (the merging part), but, deleting a whole index is much more lightweight. On Sat, Nov 12, 2011 at 3:57 AM, searchersteve [stevesuo@gmail.com](mailto:stevesuo@gmail.com) wrote: > And one more last, last question........ > > Now I'm reading about the \_TTL (time to live) field and how ES > automatically > purges old docs. That seems like an even more straightforward approach than > creating a separate index for each day's worth of data. I could put all my > docs in one index that way. > > Is THIS the optimal purge solution? I know this treads on ground previously > discussed on the list, but there seem to be varying themes over time. > > Steve > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Limit-on-number-of-indices-tp3492500p3501545.html](http://elasticsearch-users.115913.n3.nabble.com/Limit-on-number-of-indices-tp3492500p3501545.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:48am UTC](https://discuss.elastic.co/t/limit-on-number-of-indices/5825/10 "2017-07-06T03:48:55Z")