# Elasticsearch halting when multiple indices are created
**URL:** https://discuss.elastic.co/t/elasticsearch-halting-when-multiple-indices-are-created/57715
**Category:** Elasticsearch
**Created:** [August 10, 2016, 2:59pm UTC](https://discuss.elastic.co/t/elasticsearch-halting-when-multiple-indices-are-created/57715 "2016-08-10T14:59:11Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@bstascavage](https://discuss.elastic.co/u/bstascavage)
#### Post date: [August 10, 2016, 2:59pm UTC](https://discuss.elastic.co/t/elasticsearch-halting-when-multiple-indices-are-created/57715/1 "2016-08-10T14:59:11Z")
I have a cluster of 3 master nodes and 10 data nodes, 31k shards. We use this as a centralized logging platform supporting multiple products (53, to be exact), with logstash creating an index-per-day-per-product. (ie. foo-YYYY.MM.DD, bar-YYYY.MM.DD, etc). Everyday at UTC midnight, Logstash starts indexing into the new indices, which of course creates the new index, allocates the shards, etc. The problem is that at UTC midnight 53 new indices are created all at once, which is turning into an expensive process. We are seeing that indexing straightup halts during that time, sometimes up to 10-15 minutes, which breaks some automation, generates false alerts, etc.
Is there a way to stage index creation, so that all 53 indices are not all created at the exact same time? We would like to keep having an index-per-day, if possible, as it works well for searching and archiving purposes.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [August 10, 2016, 8:47pm UTC](https://discuss.elastic.co/t/elasticsearch-halting-when-multiple-indices-are-created/57715/2 "2016-08-10T20:47:45Z")
> [@bstascavage](#):
>
> 10 data nodes, 31k shards
That's massive, and waaaaaay to many and likely to be causing the issue.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 10, 2016, 9:01pm UTC](https://discuss.elastic.co/t/elasticsearch-halting-when-multiple-indices-are-created/57715/3 "2016-08-10T21:01:43Z")
Which version of Elasticsearch are you on? In Elasticsearch 2.x delta cluster state updates were introduced, which should help if you are already not using it. Before that the entire cluster state (which could be quite large given the large number of shards in the cluster) need to be propagated to all nodes for every update.
---
### Author:  [@bstascavage](https://discuss.elastic.co/u/bstascavage)
#### Post date: [August 10, 2016, 9:29pm UTC](https://discuss.elastic.co/t/elasticsearch-halting-when-multiple-indices-are-created/57715/4 "2016-08-10T21:29:05Z")
I mean, in every other capacity our node/shard ratio isn't an issue, and as I said, this wouldn't be a problem if index creation didn't all happen at once.
And this is version 2.3.3, for reference.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [August 10, 2016, 9:31pm UTC](https://discuss.elastic.co/t/elasticsearch-halting-when-multiple-indices-are-created/57715/5 "2016-08-10T21:31:07Z")
Right, but if you have **310000** shards and then you go to add **more** , ES needs to calculate where these need to go and then allocate them. That's really expensive.
How large are these shards on average?
---
### Author:  [@bstascavage](https://discuss.elastic.co/u/bstascavage)
#### Post date: [August 10, 2016, 9:33pm UTC](https://discuss.elastic.co/t/elasticsearch-halting-when-multiple-indices-are-created/57715/6 "2016-08-10T21:33:58Z")
You have an extra zero on that 😉 It's 31,000 shards, not 310,000.
A rough estimate of the shard size is maybe 5 mb?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [August 10, 2016, 9:35pm UTC](https://discuss.elastic.co/t/elasticsearch-halting-when-multiple-indices-are-created/57715/7 "2016-08-10T21:35:00Z")
Fair call, it's still way too much and your wasting massive amounts of resources to maintain that.
A 5MB shard and a 50GB shard take the same amount of resources to have open in ES.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:28pm UTC](https://discuss.elastic.co/t/elasticsearch-halting-when-multiple-indices-are-created/57715/8 "2017-07-05T22:28:39Z")