Wrong zone used for query?

[klahnakoski (Kyle Lahnakoski)]

How do I ensure my queries are directed to nodes in the same zone?

My queries are running slow, and I just learned about "profile": true,. In the profile I could see the slow [node][index][shard] that was causing the problem. It turns out the node was in another zone under heavy ingestion load (and maybe query load too). I did not expect this.

My cluster has two zones primary and spot. The primary zone has 4 nodes, and all the shards and is under heavy load. The spot zone has 40 the nodes, with the same number of shards, so less shards per node. Finally, there is one coordinator node, in the spot zone, that accepts all query requests, and has no data. In theory the coordinator should be sending queries to the spot zone, not the primary zone.

All nodes have the awareness attribute set

cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.enable: none

and all nodes have node.attr.zone set to either primary or spot as per above.

I am wondering if cluster.routing.allocation.enable: none may be causing the problem: Since it shares a prefix, maybe cluster.routing.allocation.awareness is also turned off?

Thank you

[klahnakoski (Kyle Lahnakoski)]

Of course I could not see the problem until I asked the question.

There is a daemon that was removing the awareness.attributes, and therefore disabling the query routing.