We have an Elasticsearch cluster of 7 servers in Amazon, two clients, three master nodes (with data), and two data.
All master and data nodes are m4.2xlarge with 32GB of RAM 8 cores and SSD. The 2 clients are m4.xlarge with 4 cores and 16Gb of RAM. The Elasticsearch service has 15GB of memory (half of the maximum memory) with no swap.
We have three indexes, one of the most important have 100GB of data and every index have 5 shards distributed in the 5 datas (masters included) and everything replicated correctly with all the nodes.
Our frontend is using the cluster pointing to one client, and we are getting searches around 2-3 seconds that it's acceptable for us. The client seems to distribute our queries correctly, we are using the paramedic plugin to see the balancing https://github.com/karmi/elasticsearch-paramedic
The problem begins when we start the indexation process (we index using a bulk request of 2000 documents), then we are getting searches around 17 seconds. At the same time every cluster node have 0.5 of maximum load average.
What we tried:
- Increasing index interval actually to 30s.
- Checking thread pools, but on the status of the threads _nodes/stats/thread_pool?pretty I can't see anything problematic.
Anyone has any ideas of what could happen, or what we can check/change to address this issue?
Is there any property that could limit the performance of searches when we are doing indexation at the same time?
Is our current cluster properly configured or well defined?
Thanks in advance