Hi David,
Thanks a lot for the quick response and pointing me to the right direction! Much appreciated
The test machines have lvm created with 8 local SSD disks. There was an issue with the LVM setup, therefore when elasticsearch instance was trying to restore the data onto the LVM partition, only one out of the eight attached disk was utlized at a time.
When I performed standalone disk throughput test by issuing multiple gstuil commands and achieved 600MB/s, somehow multiple local SSD disks were used. I can verify that by running iostat.
After fixing the lvm setting related to disk striping and rebuilding the host machine, I am now able to see the much improved performance.
Ray