Hi all,
We've set a new elasticsearch cluster to store skywalking data. Settings are as following:
6 data nodes, each 16GB RAM (around 50% heap size), 16 cores, 4*375G local SSDs disks (thats about 1.5TB each)
3 master nodes 4GB RAM, 4 cores, 50GB SSD persistent disk
we used to run the data nodes with pd-ssd PVC each with 2 TB, we used to get reasonable performance but with some set backs, now we switch as i stated before to Local SSDs on the node from GKE (those are set in raid0 using GKE daemonset that formats the disks in ext4 and sets them up with raid0; its provided by GCP), now every doc and suggestion indicates that the performance should be alot better, but what we are seeing is that the performance is way way worse that the SSD PVC, the initial troubleshooting tells us that the IO on these local SSDs are super bad.
Anyone have any suggestion on whats going on ?