Hello.
I have a single ES node running on my server with lots of indexes and data. It is time for me to cluster my data/indices.
Where can I learn about,
- how to size the cluster ? ( No. of nodes, shards etc keeping in mind the future volumes of data )
- how to select hardware for such a setting from No. 1, so that I can move from single node to multiple nodes for better throughput and reliability.
In my case, I have a system about 10 TB of data spread across 50 indices on a single node. The documents have 10-20 fields with some data being as large as 5MB but usually it's very small in size, i.e less than 500kB of data. Current hardware is 32 cores CPU with 64 GB RAM on NVMe. My IOPS and Disk read is enormously high because my requirement is 24/7 read/write to indices and they are not time-series, i.e any document may be updated later and also needs to be available 24/7.
Please note that I am more keen to set up cluster on-premises than using cloud services.