I have an elastic cluster (7.4).
48 core CPU, 380GB memory, SSD Disk, 2 nodes
Total data: 3TB/per node
Heal 32GB
thread_pool.search.size: 50
thread_pool.search.queue_size: 200
indices.memory.index_buffer_size: 10%
The very first time(Or after restart), I tried to generate some search workloads. For the first few mins the CPU was normal but the IOPS are high. Once the data loaded into the memory then IO bandwidth is very very less, but CPU started consuming 100%.
Hot thread:(truncated)
100.6% (503.1ms out of 500ms) cpu usage by thread 'elasticsearch[node-1][search][T#39]'
9/10 snapshots sharing following 32 elements
com.amazon.opendistroforelasticsearch.knn.index.v1736.KNNIndex.queryIndex(Native Method)
com.amazon.opendistroforelasticsearch.knn.index.v1736.KNNIndex.access$000(KNNIndex.java:34)
com.amazon.opendistroforelasticsearch.knn.index.v1736.KNNIndex$2.run(KNNIndex.java:82)
com.amazon.opendistroforelasticsearch.knn.index.v1736.KNNIndex$2.run(KNNIndex.java:80)
java.base@13.0.1/java.security.AccessController.executePrivileged(AccessController.java:753)
java.base@13.0.1/java.security.AccessController.doPrivileged(AccessController.java:312)
com.amazon.opendistroforelasticsearch.knn.index.v1736.KNNIndex.queryIndex(KNNIndex.java:79)
com.amazon.opendistroforelasticsearch.knn.index.KNNWeight.scorer(KNNWeight.java:101)
app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:181)
app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:163)
app//org.elasticsearch.search.internal.ContextIndexSearcher.searchInternal(ContextIndexSearcher.java:190)
app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:174)
app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:440)
app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:270)
all are the same queries, executed via script to perform a small load test.