Elasticsearch site provides a fairly minimal HW recommendations regarding on what to use to best utilize ES ("use SSDs), and provides no estimates on what performance should be expected according to the defined HW and i hoped that i could get recommendations on what we should use on my current setup.
currently we are generating about 2.5-3TB of data/day (about 2B documents per day) and I save it for a 7 days.
my HW specs are as follows:
3 elastic nodes of the following -
64 GB RAM (30.5 for java heap, rest for OS as per elastic recommendations).
10K spinning disks (no raid) .
4 client nodes streaming data using bulk API.
ES is 1.7.2
normal indexing work can keep up, but when running queries on the data using kibana after some time i'm starting to see a lot of GC WARN messages (GC messages every 15 minutes or so), and when i start seeing '[FIELDDATA] Data too large' errors the cluster hangs and stops responding on GC goes into overdrive.
also i am seeing alot of 'now throttling indexing: numMergesInFlight=6, maxNumMerges=5' messages, and in HQ Field Evictions number keeps growing.
first of all I want to know if its even possible for my current setup to handle such traffic?
I know that the disks are a big bottleneck here but i want to know is there something to be done except upgrading storage? and is there a way to test how much indexing rate my current setup is able to handle?
If you need additional information please let me know.