Generic ilm policy for hot-warm-cold architecture, and a generic number of nodes in a cluster

Elastic Search
1

Hi, I have some specific questions.

  1. What is the best starting point for ilm policies in a hot-warm-cold architecture? Like, what is the typical rollover values for the hot phase, and the typical retention days for the warm and cold nodes?

  2. What is the least number of nodes in a cluster with a hot-warm-cold architecture?

I'm guessing at least 2 nodes for each role (hot, warm, cold), at least 2 master nodes, and at least 1 ingest node

Thanks in advance.

2

I would say that this depends on the number of factors:

  • The number of indices you will be actively be indexing into and their expected data volume over time.
  • The overall retention period of these indices.
  • The type of hardware you will be deploying the cluster on. What is the specification of the different node types?

You always want 3 master eligible nodes, never 2. As the hot nodes will be doing all indexing you may want to make them ingest nodes as well. If you have a lot of pipelines it can make sense to create dedicated ingest nodes to which you send all bulk requests. If you do so you should make sure you have at least 2 to avoid a single point of failure.

4

For the hot nodes we were thinking about using the default rollover values, which is 30 days and 50GB i think before rolling over.

5

Hi, and thank you for your answer.

Type of hardware in total for the master nodes:

4 CPU
64 GB RAM
~10 TB SSD

For the hot nodes:
4 CPU
64 GB RAM
~10 TB SSD

Warm nodes:
4 CPU
64 GB RAM
~10 TB SSD

Cold nodes:
4 CPU
64 GB RAM
~10 TB SSD

Ingest?
4 CPU
64 GB RAM
~10 TB SSD

As of today we have around 8 pipelines. Is that a good reason to create dedicated ingest nodes, or make the hot nodes ingest as well?

6

The main idea behind hot-warm or hot-warm-cold architectures is to allow different node types to have different specialized specifications. If all nodes have the same specification I do not think you will be able to benefit from such architectures and probably would be better of deploying a cluster without zoning. Please have a look at this old blog post on the topic as it explains the roles of the different tiers.

This is overkill. Dedicated master node are typically small and do not require a lot of storage. 2 CPU cores and 8GB of RAM with 100GB of storage is probably enough. These nodes should be allowed to manage the clsuter and not be involved in request processing.

The hot nodes handle all the indexing, so often need a lot of CPU. 4 CPU cores sound low. As indexing is very I/O intensive they typically do not hold a lot of data that they would need to serve queries for. Instead they usually quite quickly hand over data to the warm tier when indices become read-only after rollover. 10TB of SSD therefore seems excesive as you may not be able to use all of it. Also note that these nodes require very high I/O so performance equal to that of local SSDs is recommended. Storage arrays or networked storage based on SSDs may not perform at the expected level.

Dedicated ingest nodes (if required) do not store data, so do not require a lot of storage. 10TB of SSD is therefore unlikely to be used. Running ingest pipelines can also be very CPU intensive, so 4 CPU cores might be a bit low.

8

Aha, I see, i'll take a look at the blog post. We are probably going to go over to the Enterprise license as well in case we need to scale up the cluster.

So, each master node should have

  • At least 2 CPU
  • 8 GB RAM
  • 100 GB Storage

Each hot nodes should have

  • At least 6 CPU?
  • 64 GB RAM (Maybe too excessive as well?)
  • Less than 10 TB, maybe 5 TB (do not use SSD)

I guess 64 GB RAM for warm and cold is too excessive as well.
But the storage may be realistic?
Is 4 CPU enough?

For the ingest nodes, I agree with the point about storage, and should be way less.
I'm guessing around 6 CPU here as well.
And around 64 GB RAM or less.

9

Looks reasonable. make sure not to send requests to these noddes.

I would have at least 8 CPU cores.

The nodes require local SSDs as they are the most I/O intensive. I would keep 64GB of RAM.

64GB is not necessarily excessive, but it will depend on your data and queries. As these nodes will hold a lot of data it will be your query latency requirements that decides how much RAM and CPU is required.

11

Ok, so it's just a matter of setting the values we discussed now for the hardware and test it out and change them if needed. Thank you for your input, it has been really helpful!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.