Hot/Warm/Cord Architecture

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

I'm trying to figure out what are the most optional ratios for each data storage. I'm aware about 30/100 ratios used for hot/warm storages in Elastic Cloud. I can't really find much information about the cold storage.

  1. What would be the recommended ratio for the cold storage in ECE? If I increase it, let's say to 120 and it's running on a slower hardware, would this downgrade performance to the point that my searches would take very long time to return? How searchable would be the data in the cold storage?
  2. If I use SSD drives for the hot storage, how much can I increase the hot storage ratio and still have solid performance? I totally understand that this will depend on specific requirements for each customer.

I'm trying to see how much performance downgrade/upgrade with the ratio change. Is this something I can easily measure before making final decisions?

--Thansk

2

Currently we're advising people to be very careful with cold architectures on ECE. The problem is that ECE treats the data storage of each node as transient and uses the native ES replication to move data around. If you have huge storage ratios on cheap hardware then moving one node eg following hardware failure might become prohibitively expensive.

As a result of the above we haven't done any analysis of cold-on-ECE, and to my knowledge there is nobody actively using it who could provide real-world information. In general (and aside from the replication problem), all the usual performance characteristics of hot-warm-cold as "standard" Elasticsearch deployments should hold (and probably there are folks in the elasticsearch forum who could give numbers that we don't have?)

Alex

1 Like
3

We are planning to add the Cold Architecture for our Enterprise install of Elastic stack that we run on our AWS account to store and search close to 180TB of data (with replica included) kept for 90 days. We currently have the Hot and Warm We want to keep 7 days of data in Hot (SSD - 10 nodes 5 TB each), 7-50 days in Warm (HDD - 10 nodes 12 TB each) and 50-90 days in Cold (currently thinking 2 nodes HDD -d2.4x large with 25TB disk each with 80% of it holding data). Data will be moved through ILM. The replication priority setting for Cold shards will be set low so it does not affect the cluster. We have the Hot and Warm working in production fine for a year now, adding Cold now to increase days of retention. Do you see any problems with this architecture espl. the Cold nodes? @Alex_Piggott

4

Does a deployment in cloud help with hardware failure?

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.