We have a test which has so many writes to elastic search and also searches
Hardware Specification :
8 crores
32 GB RAM(16GB to ES Heap Size)
RAID 0 Disk
Test Conditions :
- Totally 9 indexes are involved in this test
- Bulk updating of 1 million documents on each of the 5 indexes and each having 1 million documents
- 3000 updates on index per hour and 1500 searches on the same index
- 100 updates on each of other 3 indexes
Just gave you test overview so that you an idea of volume of searches vs indexes
With the default settings of thread pool for bulk operation we see that upgrade transactions are pretty slow 1 doc per 29 seconds on average .
Based on the learning from web we found that elastic designed to give priority to search that is based on elastic search bulk and search default thread configurations .
With default of bulk (which is what update uses) thread pool configuration , we have only 8 threads on 8 cores which does updates /inserts in elastic and I can clearly see that 8 threads is not enough for this test
So I changed the bulk threads =60 and ran the same test which shown 90% improvement in updates and 50% degradation in searches
threadpool.bulk.type: fixed
threadpool.bulk.size: 60
I changed the no of threads for bulk operation but why does the search operations degraded ?
What is the relation between bulk and search thread pool ?
If disk is the bottle neck as per previous tests why does this setting helps to improve the updates speed by 50 %
Also
how to make particular node as a update centric ?
how to make particular node as a search centric ?