My elastic search cluster contains 3 nodes running JVM: 1.8.0_144 ES: 6.1.2
My nodes information is below:
I created a index with 12 shards with replication of 0 and refresh_interval -1 and index.translog.durability ASYNC. I loaded 54M documents into elasticsearch cluster via spark job.
The job completed in 90 min. I changed the index property to
number_of_replicas = 1, refresh_interval = 1s and translog.durability to REQUEST
The cluster turned yellow and after 2 hrs changed to green. I see all shards replicated and everything looks good. However the CPU on the hosts has not come down yet even after several hours. I see an average of 50% used on all 3 nodes.
Attached is the hot_threads output. Can someone tell me how to debug the issue? Is this a known problem or I am missing some configuration?