Hello,
Based on what i have read looks like all shards in a cluster have all percolation queries for a given index ?
I have one index and 15 shards (+1 replica - total 30 shards) -- and only percolation queries in that index.
I dont store any documents in ES. I create a document and run through the percolate API to get the matching queries. I would have thought based on https://www.elastic.co/guide/en/elasticsearch/client/java-api/1.7/transport-client.html that the load on each node in the cluster would be same. that does not seem to be the case. There is one node which has very high cpu usage.
This leads me to believe that percolation queries are split across different shards and to get the results, es internally runs in parallel the percolation on all shards to collates them to return to user, hence the above scenario would be explained if one of the queries is very complex or a set of complex queries happen to be on the same shard (and some set of shards are on the same physical node) ?
even in the above case i would think there would be two shards that will have high cpu usage one primary and one replica, why is this not happening ?
Given the above how do i handle cases where some queries are more complex and hence more cpu intensive than others?
the cluster is over provisioned to have 9 nodes and one index with about 40 percolation queries doing geo distance filter with about 174000 lat lon across all queries.
if any other details are required please do let me know. Thanks!