issue: our cluster is observing high read operations
elasticsearch version: elasticsearch-6.2.1
5 nodes each with 2 cpus and 7GB or ram
the data volume for elasticsearch uses a zfs pool mounted using an ssd drive and the scheduler for that drive is set to noop
here is the node stats
https://pastebin.com/ThgJjX5u
here is the hot threads
https://pastebin.com/7Rv2vzGa
i have also uploaded a diagram of the IO from a box and notice the avg reads per seconds is around 400.