# Does a lot of indices in elasticsearch cluster have an impact on performace of elasticsearch

**URL:** <https://discuss.elastic.co/t/does-a-lot-of-indices-in-elasticsearch-cluster-have-an-impact-on-performace-of-elasticsearch/58544>\
**Category:** Elasticsearch\
**Created:** [August 22, 2016, 2:54am UTC](https://discuss.elastic.co/t/does-a-lot-of-indices-in-elasticsearch-cluster-have-an-impact-on-performace-of-elasticsearch/58544 "2016-08-22T02:54:21Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![fatelei](https://avatars.discourse-cdn.com/v4/letter/f/bc79bd/32.png) [@fatelei](https://discuss.elastic.co/u/fatelei)\
**Post date:** [August 22, 2016, 2:54am UTC](https://discuss.elastic.co/t/does-a-lot-of-indices-in-elasticsearch-cluster-have-an-impact-on-performace-of-elasticsearch/58544/1 "2016-08-22T02:54:21Z")

</div>

I am wondering if an elasticsearch cluster has a lot of indices (may be 100 or 1000），does the elasitcsearch cluster have performance issue ?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 22, 2016, 2:55am UTC](https://discuss.elastic.co/t/does-a-lot-of-indices-in-elasticsearch-cluster-have-an-impact-on-performace-of-elasticsearch/58544/2 "2016-08-22T02:55:48Z")

</div>

It's more the number of shards that can cause issues.

---

<div class="post-metadata">

**Author:** ![fatelei](https://avatars.discourse-cdn.com/v4/letter/f/bc79bd/32.png) [@fatelei](https://discuss.elastic.co/u/fatelei)\
**Post date:** [August 22, 2016, 2:56am UTC](https://discuss.elastic.co/t/does-a-lot-of-indices-in-elasticsearch-cluster-have-an-impact-on-performace-of-elasticsearch/58544/3 "2016-08-22T02:56:44Z")

</div>

More shards with small cluster ?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 22, 2016, 3:06am UTC](https://discuss.elastic.co/t/does-a-lot-of-indices-in-elasticsearch-cluster-have-an-impact-on-performace-of-elasticsearch/58544/4 "2016-08-22T03:06:49Z")

</div>

Lots of shards, irrespective of the cluster size.

---

<div class="post-metadata">

**Author:** ![fatelei](https://avatars.discourse-cdn.com/v4/letter/f/bc79bd/32.png) [@fatelei](https://discuss.elastic.co/u/fatelei)\
**Post date:** [August 22, 2016, 3:15am UTC](https://discuss.elastic.co/t/does-a-lot-of-indices-in-elasticsearch-cluster-have-an-impact-on-performace-of-elasticsearch/58544/5 "2016-08-22T03:15:27Z")

</div>

Thanks

---

<div class="post-metadata">

**Author:** ![545945](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/545945/32/8498_2.png) [@545945](https://discuss.elastic.co/u/545945)\
**Post date:** [August 22, 2016, 1:33pm UTC](https://discuss.elastic.co/t/does-a-lot-of-indices-in-elasticsearch-cluster-have-an-impact-on-performace-of-elasticsearch/58544/6 "2016-08-22T13:33:41Z")

</div>

What do you consider "Lots of shards"? Would 256 indices on ten servers with 5 shards and 1 replica qualify? That is 2560 shards. I also have a second cluster that has well over 20k shards. I agree that too many shards is a bad thing but I have never seen anything discussing the number of shard in relation to the number of indices or number of nodes / servers. I would assuming 200,000 shards on a cluster of 100 nodes is different than on 20 nodes. Thoughts?

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [August 22, 2016, 3:53pm UTC](https://discuss.elastic.co/t/does-a-lot-of-indices-in-elasticsearch-cluster-have-an-impact-on-performace-of-elasticsearch/58544/7 "2016-08-22T15:53:10Z")

</div>

> [@545945](#):
>
> Would 256 indices on ten servers with 5 shards and 1 replica qualify?

I think that should be fine. I ran ES 1.x with 1600 indexes with mostly 1 shard and 2 replicas. Ended up with something like 6000 shards on 21 servers. Cluster state updates took some time but push through but it wasn't bad. I noticed it though. In 2.x Elasticsearch got cluster state diffs which really really improved performance for large numbers of shards so I expect that to be fine.

Large numbers of shards can cause pending cluster state tasks to wait for a long time. So the thing to watch is the age of the oldest task in [this](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-pending.html) API. The actual count of tasks is fairly bursty and not really a big deal because lots of then can be processed at once.

> [@545945](#):
>
> I would assuming 200,000 shards on a cluster of 100 nodes is different than on 20 nodes.

Since cluster state is replicated to all the nodes larger cluster states are worse for larger clusters. Less bad in 2.x than in 1.x. Much less bad, actually. But it is still a think.

There are also per shard and per field overheads that only exist on the nodes hosting the shards. These don't tend to come up as a problem nearly as often when you have many indexes as cluster state maintenance.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:26pm UTC](https://discuss.elastic.co/t/does-a-lot-of-indices-in-elasticsearch-cluster-have-an-impact-on-performace-of-elasticsearch/58544/8 "2017-07-05T22:26:08Z")

</div>


