We have previously gotten 16 warm-data-nodes (running on Kuber) all having the same resource (32Gb heap, 64Gb Ram, 3 Tb hdd).
Since it is not much recommended to use a distributed storage system for Elasticsearch, we have decided to use local disks instead.
As we were running on kuber and it provided us several advantages, we were looking for a kuber storage to have a neer-local disk performance and we have found the OpenEbs(Local PV: LVM) technology to do so.
The new cluster has 10 warm nodes each having the same Ram and CPU as before, with 8 Tb of hdd storage, which we expected to at least provide tha same performace and latency as our previous cluster does.
Despite the fact that reducing the number of nodes may reduce perfomance a bit, though we think that this is not the real problem, and the root cause is the openEbs itself.
We have encountered about 100x higher latency (0.2 ms -> 20ms) and 10X higher took time than our previous cluster.
What is the problem? How to solve? Why a neer-local-disk technology causes such a reduction in performance, in contrast to the Ceph, which was not recommended much?