# How can an one node server handle shards limit?

**URL:** <https://discuss.elastic.co/t/how-can-an-one-node-server-handle-shards-limit/370855>\
**Category:** Elasticsearch\
**Created:** [November 20, 2024, 5:28pm UTC](https://discuss.elastic.co/t/how-can-an-one-node-server-handle-shards-limit/370855 "2024-11-20T17:28:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Benny-RR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benny-rr/32/145137_2.png) [@Benny-RR](https://discuss.elastic.co/u/Benny-RR)\
**Post date:** [November 20, 2024, 5:28pm UTC](https://discuss.elastic.co/t/how-can-an-one-node-server-handle-shards-limit/370855/1 "2024-11-20T17:28:14Z")

</div>

Hello, I have been assiged to maintain an ELK stack server where there are scripts that run automatically, these scripts get reports from IntelMQ and VulnWhisperer. IntelMQ's reports generate an index per day and the VulnWhisperer's reports once a month, but everytime a new index is created it makes two shards, the primary one and one for the replica.

The main problem our infraestructure is only an one node ELK server and some few month back I encounter with this error:

"Validation Failed: 1: this action would add [2] shards, but this cluster currently has [999]/[1000] maximum normal shards open"

And the temporal solution was increase the maximum normal shards open to 1500, I read about shards and raising the maximum number of shards is not recomended and some of the best practices like "add more nodes" are not an option for now, while other options like merging old indices or those were I no longer write to them... Can I merge all IntelMQ daily indices from 2023 into a single one? And if that's possible, How can I do that?

Or there is a better option for one node server to handle this problem without increasing the maximum shard limit?

My version of Elasticsearch is 8.12.1

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 20, 2024, 6:32pm UTC](https://discuss.elastic.co/t/how-can-an-one-node-server-handle-shards-limit/370855/2 "2024-11-20T18:32:32Z")

</div>

Hello and welcome,

> [@Benny-RR](#):
>
> IntelMQ's reports generate an index per day and the VulnWhisperer's reports once a month, but everytime a new index is created it makes two shards, the primary one and one for the replica.

What is the average size for the daily indices created and for the monthly ones?

Also, you do not have replicas, so if the monthly indice is creating 2 shards, they are 2 primary shards, not replicas. It is not possible to have replicas with a single node.

> [@Benny-RR](#):
>
> Can I merge all IntelMQ daily indices from 2023 into a single one? And if that's possible, How can I do that?

Yes, you can do that using the [reindex api](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/docs-reindex.html).

It would be something like this:

```auto
POST _reindex
{
  "source": {
    "index": "indexname-2023.*"
  },
  "dest": {
    "index": "indexname-2023"
  }
}

```

Keep in mind that this will require that you have free space to store the data from the daily indices and the data for the yearly one during the reindex process.

---

<div class="post-metadata">

**Author:** ![Benny-RR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benny-rr/32/145137_2.png) [@Benny-RR](https://discuss.elastic.co/u/Benny-RR)\
**Post date:** [November 20, 2024, 7:31pm UTC](https://discuss.elastic.co/t/how-can-an-one-node-server-handle-shards-limit/370855/3 "2024-11-20T19:31:34Z")

</div>

Hello and thank you for your response!

Answering the first question, the average size is around 1-2MB per index daily.

About the 2 shards, one is set as a primary shard and the other one as replica but this second shard goes as "UNASSIGNED" so we have one assigned shard, the primary one, and the unassigned one, the replica one. About this problem, how can I overcome it?

I will try using the reidex api, thank you so much. The storage is not an issue for now.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 20, 2024, 8:01pm UTC](https://discuss.elastic.co/t/how-can-an-one-node-server-handle-shards-limit/370855/4 "2024-11-20T20:01:05Z")

</div>

> [@Benny-RR](#):
>
> Answering the first question, the average size is around 1-2MB per index daily

With such smaller shard size there is no reason to have daily index, it can be pretty inneficient and lead to the issues you are having with the number of shards.

The recommend shard size is something around 40 GB.

You could switch to yearly indices, or if you want to keep daily indices it would be better to reindex the daily indices into a yearly one once a month has passed.

> [@Benny-RR](#):
>
> About the 2 shards, one is set as a primary shard and the other one as replica but this second shard goes as "UNASSIGNED" so we have one assigned shard, the primary one, and the unassigned one, the replica one. About this problem, how can I overcome it?

This is expected as you have a single-node, a replica shard cannot be allocated to a node where its primary is already allocated, so your replica will be unassigned and your cluster status will be yellow.

These replica shards would only be allocated if you added another node into your cluster.

To solve this for a single-node you need to configure your indice to not have replica shards, this is done in the template by setting `index.number_of_replicas` to `0`.

This can be done dynamically with the following request in Kibana Dev Tools:

```auto
PUT /indexname/_settings
{
 "index" : {
  "number_of_replicas":0
 }
}

```