ES node disk sizing

[Benjen]

I am studying on ES node storage and would like to have an idea on what is the best practice/recommendation hardware config for a ES node, and what is the capability of it. Yet, got confused on that.

Based on my understanding on the information from Elastic, considered to deploy a single on-premise ES node in this environment:

• Daily log size: 6GB
• Log retention: 6 months
• Shards frequency: Daily

From this,

• estimated no. of shards will be 31 x 6 = 186 shards and,
• total size will be 186 x 6 = 1116 GB

Based on this, the ES node should configured with:

• Heap size >= 5.9 GB (186 / 20)
  https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster
  “A good rule-of-thumb is to ensure you keep the number of shards per node below 20 per GB heap it has configured”

• RAM >= 12 GB
  https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
  “Set Xmx ( heap size ) to no more than 50% of your physical RAM”

As a conclusion, the ES node I am going to have should configured with at least:

• CPU = 8 core
• RAM = 12GB
• Storage = 1116GB

Does anyone know if I am estimating the machine size correctly or not? Many thanks!

Other ref:
https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_memory
A machine with 64 GB of RAM is the ideal sweet spot

[Christian_Dahlqvist]

The amount of heap you need for a certain data volume is not just driven by shard count, but also your data, mappings and how well you have optimized these. You have also assumed a 1:1 ration between raw data and indexed size on disk. This can depending on mappings go higher or lower, and as you never want to get too close to a completely full disk I would probably recommend adding a bit more storage, e.g. 10% - 15%.

Given you are indexing relatively little data per day I also suspect you will not need quite that many CPU cores. 4 might be enough unless you are going to run very heavy queries. Heap and RAM sounds reasonable.

[Benjen]

@Christian_Dahlqvist

The amount of heap you need for a certain data volume is not just driven by shard count, but also your data, mappings and how well you have optimized these.

Thanks for the reminder, for sure there are numerous factors which affect the performance of an ES node and worth to consider.

I am also wondering, does Elastic have any ES node hardware configuration best practice that should follow? (eg. disk size should not exceed xTB / RAM should not exceed x GB..........)

Please just assume that I can provide unlimited resources for that single ES node scenario. I understood having ES cluster with multi nodes definitely a better choice in this case.