# What is the best way to distribute nodes and shards in Elasticsearch to achieve fast search while storing recent data on SSD and older data on HDD?
**URL:** https://discuss.elastic.co/t/what-is-the-best-way-to-distribute-nodes-and-shards-in-elasticsearch-to-achieve-fast-search-while-storing-recent-data-on-ssd-and-older-data-on-hdd/380988
**Category:** Elasticsearch
**Created:** [August 13, 2025, 6:57am UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-distribute-nodes-and-shards-in-elasticsearch-to-achieve-fast-search-while-storing-recent-data-on-ssd-and-older-data-on-hdd/380988 "2025-08-13T06:57:59Z")
**Posts on this page:** 8
**Page:** 2
### Author:  [@Ella\_conan](https://discuss.elastic.co/u/Ella_conan)
#### Post date: [August 14, 2025, 9:30am UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-distribute-nodes-and-shards-in-elasticsearch-to-achieve-fast-search-while-storing-recent-data-on-ssd-and-older-data-on-hdd/380988/21 "2025-08-14T09:30:16Z")
Okay, I will follow your advice.
Thank you both for your help and for the quick response
---
### Author:  [@RainTown](https://discuss.elastic.co/u/RainTown)
#### Post date: [August 14, 2025, 9:52am UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-distribute-nodes-and-shards-in-elasticsearch-to-achieve-fast-search-while-storing-recent-data-on-ssd-and-older-data-on-hdd/380988/22 "2025-08-14T09:52:44Z")
> [@Ella\_conan](#):
>
> Yes, a single server with the following specs:
>
> - 1.5 TB RAM
> - 144 CPU
> - 75 TB SSD
Great advice from
@Christian_Dahlqvist I add only to confirm that server will be 100% dedicated to elasticsearch ? Your initial plan only allocated 320GB of it for 5x hot nodes?
And second, 75TB of local SSD storage is a lot. But equally important here is the aggregate IOps/bandwidth you can realistically achieve. Best if you have as many independent paths to the storage as VMs.
Lastly, single server solution is obviously no redundancy. Consider you “server on fire” scenarios.
---
### Author:  [@Ella\_conan](https://discuss.elastic.co/u/Ella_conan)
#### Post date: [August 14, 2025, 1:22pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-distribute-nodes-and-shards-in-elasticsearch-to-achieve-fast-search-while-storing-recent-data-on-ssd-and-older-data-on-hdd/380988/23 "2025-08-14T13:22:25Z")
Yes, initially this will be 320 GB, but what do you recommend? I can increase it to 128 GB per node, or even divide the 1.5 TB among the five nodes.
The server has 24 drives of type KR-09N32F-SSW00-468-00LM-A02 — you can look into this.
Yes, you are correct; if the entire server burns out or fails, I have a DR plan in place.
---
### Author:  [@RainTown](https://discuss.elastic.co/u/RainTown)
#### Post date: [August 14, 2025, 2:22pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-distribute-nodes-and-shards-in-elasticsearch-to-achieve-fast-search-while-storing-recent-data-on-ssd-and-older-data-on-hdd/380988/24 "2025-08-14T14:22:00Z")
> [@Ella\_conan](#):
>
> Yes, initially this will be 320 GB, but what do you recommend? I can increase it to 128 GB per node, or even divide the 1.5 TB among the five nodes.
I recommend giving almost all the memory to elasticsearch VMs. @Christian_Dahlqvist suggested starting to test at 10x, and I’ve no reason to argue with that. How useful is leaving memory unallocated, letting host OS use it for caching? My hunch is “not very”.
On the disks , you need know about the paths to the disks. 24 disks are across how many controllers? Actually, there 101 ways you could map your dish’s to VMs. Does your host and VM OS need to live on same disks? If so, maybe assign 2 for that, which leaves 22, which is 2x11. 24 is a bit awkward to divide by 10.
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [August 14, 2025, 7:24pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-distribute-nodes-and-shards-in-elasticsearch-to-achieve-fast-search-while-storing-recent-data-on-ssd-and-older-data-on-hdd/380988/25 "2025-08-14T19:24:31Z")
I have SSD/NVME on proxmox and if I use ceph or even map drive via proxmox to vm it is slow, very slow.
After lot of testing I have found that only direct pass through disk to VM works best and gets me speed that is describe by vendor.
this means you can’t fail over vm. for example one of the VM. and scsi2 is my data disk, scsi0 is OS
---
### Author:  [@Ella\_conan](https://discuss.elastic.co/u/Ella_conan)
#### Post date: [August 16, 2025, 3:19pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-distribute-nodes-and-shards-in-elasticsearch-to-achieve-fast-search-while-storing-recent-data-on-ssd-and-older-data-on-hdd/380988/26 "2025-08-16T15:19:58Z")
> [@RainTown](#):
>
> On the disks , you need know about the paths to the disks. 24 disks are across how many controllers? Actually, there 101 ways you could map your dish’s to VMs. Does your host and VM OS need to live on same disks? If so, maybe assign 2 for that, which leaves 22, which is 2x11. 24 is a bit awkward to divide by 10.
There is only one controller, and all the data functions as a single hard disk. I used RAID 6 for this setup.
---
### Author:  [@Ella\_conan](https://discuss.elastic.co/u/Ella_conan)
#### Post date: [August 16, 2025, 3:21pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-distribute-nodes-and-shards-in-elasticsearch-to-achieve-fast-search-while-storing-recent-data-on-ssd-and-older-data-on-hdd/380988/27 "2025-08-16T15:21:41Z")
In my case, there are no physical machines. I am creating virtual machines from the server, dividing it into 5 virtual machines.
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [August 17, 2025, 7:52pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-distribute-nodes-and-shards-in-elasticsearch-to-achieve-fast-search-while-storing-recent-data-on-ssd-and-older-data-on-hdd/380988/28 "2025-08-17T19:52:12Z")
then nothing much I guess you can do. run some benchmarking for example run dd for read and write
one dd, then 5 dd and then may be 20 and see average speed it does
for example:
dd if=/dev/random of=/datafilesstem/ella1.test count=10000000
[Previous page](https://discuss.elastic.co/t/what-is-the-best-way-to-distribute-nodes-and-shards-in-elasticsearch-to-achieve-fast-search-while-storing-recent-data-on-ssd-and-older-data-on-hdd/380988.md?page=1)