Recommended node hardware/settings for warm data

Elastic Stack Elasticsearch
1

Hello,

I am looking at an allocation strategy to move around 2800 indices that are mostly idle with few reads per week to slower nodes and with this, clean a bit of space in our high-performance nodes. These indices accounts for 5.6K shards and around 100TB of disk space.

The configuration for the allocation is well documented and for our cluster version 7.4, we would need to work with index.routing.allocation.(include,exclude,require) options.

But I didn't see anything on recommendations for hardware for warm or cold nodes, especially if we check the current recommendation of 20 shards per GB and the 32GB JVM limit, this kind makes a bit hard to estimate moving this data to bigger instances with more memory and HD storage.

Does anyone have any tips or documentation that can share? I've been googling this for a couple of weeks, but no luck so far.

I thought about merging or shrinking the indices but would demand changes on our applications, so I wanted to check here first.

2

Hi @Maycon_Santos

A great place to start on those recommendation is that we publicly publish the specs we use for our hosted offering... so you can pick your favorite cloud provider and we show what instances we use and you can map that back to your HW specs.

Here is the overview

And Example here are details on for AWS specs

Also when you described "Mostly Idle" that sounds a lot like Cold or even Frozen which is Amazing.

You will still want to be Limited to the Correct JVM sizes unless you are going to go MUCH larger...

What we have seen with some larger Basic / OSS / Free implementations , that moving to the Commercial Model with Searchable Snapshots (a paid for feature) with Frozen nodes can actually reduce your overall cost because it dramatically reduces your HW footprint (apologies not trying to sell but I come from an Enterprise Arch background more for less was usually a good things)

Frozen nodes use Object Store as a backing data store, and can easily support 1600:1 or ~100TB of S3 storage per Frozen Node.

This is all GA now...

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.