How do I calculate the number of shards I should have?

Elastic Stack Elasticsearch
1

Hello

Following the advice given on this forum (thank you), I have been able to reduce from August to now, shard amount: From 4028 to 3317

That being said, I feel this is a high number.

There is a document that says how to calculate what the ideal size should be but I cant seem to find it and if I remember, its kind of a few years old (I might be mistaken)

Could someone help me calculate this?

Thank you

2

May I suggest you look at the following resource about sizing:

In short:

  • Something like 20 to 50gb par shard
  • 20 shards per gb of HEAP
1 Like
3

Hello

I forgot to mention this is all running on one node.

I dont quite fully understand the article I think.....Il reread it.

In short:

  • Something like 20 to 50gb par shard
  • 20 shards per gb of HEAP

Since I changed over to monthly, my biggest index (1 shard) is 12Gb.

But my heap I think is 16GB so (16 * 20 = 320), I still have to lower it.

4

You have to lower the number of shard indeed.
And may be reindex old daily indices for example into monthly indices.

Better could be to use https://www.elastic.co/guide/en/elasticsearch//reference/current/index-lifecycle-management.html and https://www.elastic.co/guide/en/elasticsearch//reference/current/data-streams.html

5

I had daily indexes before and after 180 days, those dailies are being erased.

New logs are all monthly and thats why (I think) the shard amount is going down.

It will still take a while but they are indeed going down.

I actually use ILM for 180 days deletion.

Im not sure what data streams are exactly ; What are their pros and cons for logs that are constantly being sent and are time based?

6

If your cluster (or node) is under pressure and you can't reindex, you can always start to remove manually the older indices without having to wait for 180 days.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.