I would like the hardware requirements for good resiliency - Elastic 8.4
I looked on the documentation and didnt find any info pertaining to hardware specs.
We currently have 3 master nodes and 17 data nodes - 13 warm nodes and 4 hot nodes. Do we need to keep same number of warm/hot nodes or its as per our data search frequency?
Hot nodes -- 64 GB memory,storage :500 GB, heap :50% RAM
Warm nodes - 64GB MEMORY, storage of 500 GB, heap : 50% RAM.
Does the spec seem to be same for both warm and hot nodes??
What I would do is look at our published hardware specs / profiles for how we run Elastic Cloud. We would consider these best practices.
There's quite a lot of information to help you think about What would you would like for your hot and warm and even cold and frozen tiers.
And to answer your simple question, no hot and warm profiles are not usually the same.
Typically warm has much more storage than hot per node... And typically can run on cheaper storage like HDD
This is a nice page which shows ratios of CPU / RAM / Storage etc.
And you can see the example on the Elastic Cloud Pricing Page
Existing elastic is installed not on cloud env. its onprem
In that case , can you provide me the page which has the hardware info
We do not have a page for on Prem...
We say look at how we do it in Elastic Cloud and use those as your baseline HW Profiles.
Those are our recommendations.
Example Hot Node
8+ CPU 64GB RAM 1-5 TB SSD etc..
An hot-warm architecture is specifically useful when dealing with time-series data. It assumes that immutable data more or less continuously arrives in near-real time and that the most recent data is queried most frequently.
Indexing in Elasticsearch can be very I/O intensive, so the main point of a hot-warm architecture is to optimise node usage by having nodes with different hardware profiles perform different roles. As described in this old blog post hot nodes typically have very fast SSD storage as they handle all I/O intensive indexing as well as querying of the most recent data that resides on these nodes.
Once indices are no longer written to, they are optmised and moved on to the warm nodes, which only serve queries against older data. These nodes have lower load and can therefore store more data. They therefore tend to have larger volumes of slower storage than the hot nodes.
Your hot and warm nodes appear to have more or less the same storage, which does not make sense to me. What type of storage are you using on the different node types? How many CPU cores do you have allocated? What is your use case?
How many nodes you will need of different types will depend a lot on your query performance requirements as well as how much data you index per day and how long you want to keep it around. The hardware profile naturally also has an impact.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.