Elastic Search Thread Pool Configuration and interplay between search and index threads


(venkata nagaraju buddarapu) #1

We have a test which has so many writes to elastic search and also searches

Hardware Specification :

8 crores
32 GB RAM(16GB to ES Heap Size)
RAID 0 Disk

Test Conditions :

  1. Totally 9 indexes are involved in this test
  2. Bulk updating of 1 million documents on each of the 5 indexes and each having 1 million documents
  3. 3000 updates on index per hour and 1500 searches on the same index
  4. 100 updates on each of other 3 indexes

Just gave you test overview so that you an idea of volume of searches vs indexes

With the default settings of thread pool for bulk operation we see that upgrade transactions are pretty slow 1 doc per 29 seconds on average .

Based on the learning from web we found that elastic designed to give priority to search that is based on elastic search bulk and search default thread configurations .

With default of bulk (which is what update uses) thread pool configuration , we have only 8 threads on 8 cores which does updates /inserts in elastic and I can clearly see that 8 threads is not enough for this test

So I changed the bulk threads =60 and ran the same test which shown 90% improvement in updates and 50% degradation in searches

threadpool.bulk.type: fixed
threadpool.bulk.size: 60

I changed the no of threads for bulk operation but why does the search operations degraded ?

What is the relation between bulk and search thread pool ?

If disk is the bottle neck as per previous tests why does this setting helps to improve the updates speed by 50 %

Also
 how to make particular node as a update centric ?
 how to make particular node as a search centric ?


(system) #2