How can an one node server handle shards limit?

Elastic Stack Elasticsearch
1

Hello, I have been assiged to maintain an ELK stack server where there are scripts that run automatically, these scripts get reports from IntelMQ and VulnWhisperer. IntelMQ's reports generate an index per day and the VulnWhisperer's reports once a month, but everytime a new index is created it makes two shards, the primary one and one for the replica.

The main problem our infraestructure is only an one node ELK server and some few month back I encounter with this error:

"Validation Failed: 1: this action would add [2] shards, but this cluster currently has [999]/[1000] maximum normal shards open"

And the temporal solution was increase the maximum normal shards open to 1500, I read about shards and raising the maximum number of shards is not recomended and some of the best practices like "add more nodes" are not an option for now, while other options like merging old indices or those were I no longer write to them... Can I merge all IntelMQ daily indices from 2023 into a single one? And if that's possible, How can I do that?

Or there is a better option for one node server to handle this problem without increasing the maximum shard limit?

My version of Elasticsearch is 8.12.1

2

Hello and welcome,

What is the average size for the daily indices created and for the monthly ones?

Also, you do not have replicas, so if the monthly indice is creating 2 shards, they are 2 primary shards, not replicas. It is not possible to have replicas with a single node.

Yes, you can do that using the reindex api.

It would be something like this:

POST _reindex
{
  "source": {
    "index": "indexname-2023.*"
  },
  "dest": {
    "index": "indexname-2023"
  }
}

Keep in mind that this will require that you have free space to store the data from the daily indices and the data for the yearly one during the reindex process.

3

Hello and thank you for your response!

Answering the first question, the average size is around 1-2MB per index daily.

About the 2 shards, one is set as a primary shard and the other one as replica but this second shard goes as "UNASSIGNED" so we have one assigned shard, the primary one, and the unassigned one, the replica one. About this problem, how can I overcome it?

I will try using the reidex api, thank you so much. The storage is not an issue for now.

4

With such smaller shard size there is no reason to have daily index, it can be pretty inneficient and lead to the issues you are having with the number of shards.

The recommend shard size is something around 40 GB.

You could switch to yearly indices, or if you want to keep daily indices it would be better to reindex the daily indices into a yearly one once a month has passed.

This is expected as you have a single-node, a replica shard cannot be allocated to a node where its primary is already allocated, so your replica will be unassigned and your cluster status will be yellow.

These replica shards would only be allocated if you added another node into your cluster.

To solve this for a single-node you need to configure your indice to not have replica shards, this is done in the template by setting index.number_of_replicas to 0.

This can be done dynamically with the following request in Kibana Dev Tools:

PUT /indexname/_settings
{
 "index" : {
  "number_of_replicas":0
 }
}