We have built elastic search cluster for our application autosuggest and we
are facing one strange issue, need immediate help. Below is the setup of ES
cluster:
We built the azure elastic search cluster with:
Master only Nodes: 3 medium VMs with 7 GB RAM
Non-master non-data nodes: 3 medium VMs with 7 GB RAM
Data nodes: 3 large VMs with 14 GB RAM
ES version: 1.1.1
Java version: 1.7.0_45
Index size is: 1GB
Replica:1
Shards: 5
Java Heap size is half of RAM size.
In a definite pattern everyday, we see the number of requests increases
x100. I think our cluster should be able to handle that load, but 99% of
search requests goes only on one node(and everyday that node keeps
changing), and other node have only 1% traffic. Because of this, the search
thread pool queue increases on that node. If that load would have divided
on all the data nodes then we wouldn't have had any problem. I have
increased the search thread pool count to 2000 on each of data nodes. But
when requests increases on one node all threads get occupied and queue
start building. Today I mitigated it by setting the search thread pool type
to cached (that will have unbounded search thread pool).
Please let me know if anyone faced similar issue or know how to make the
load balancer work properly. We are using bigdesk to monitor the traffic,
please let me know if you need any more information.
Thanks,
Manish
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO4PsGGJkP5pwCypegbnS_ik3%3D8NVMe3b06f_3gZgQ%3DQHrpXag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.