Executing a custom allocation decider before an ES allocation decider

Elastic Stack Elasticsearch
1

The createAllocationDeciders API in ClusterModule.cs instantiates multiple ES deciders as well as custom allocation deciders. Since these are added to a linkedHashMap, they are always executed in order.

How can I ensure that my custom allocation decider executes before an ES defined allocation decider?

2

The order of allocation deciders shouldn't matter, because the final decision is the minimum of the decisions from each decider (where NO < THROTTLE < YES). Can you explain why the order is important to you?

3

From the code, it looks like, you call the AllocationDeciders.canAllocate() API.
TheAllocationDeciders is a composite of multiple decisions.

It does help short track if a NO decision is made.

In my case, I am trying to minimize the number of calls to allocationDeciders. Since my custom decider will filter out a lot of ineligible nodes, I want to call it before the AwarenessDecider.

4

I see, yes, a NO does short-circuit the later deciders. I'm a bit surprised that the awareness decider is expensive enough to need it, however. Do you know why it's slow?

Also, what is the decider you're implementing? Maybe we could run the existing deciders in a better order instead?

5

I am working on a very large cluster and want to efficiently filter out nodes which are ineligible for allocation.

The nodes in the cluster have 4-5 attributes. I am already EnableAllocationDecider and FilterAllocationDecider and ShardsLimitAllocationDecider but I also have my custom allocation decider to drastically reduce the number of eligible nodes.(it used the number of partitions in an index)

I want to call awareness decider after all the above allocators have run to ensure equal distribution across AvailabilityZones.

6

I cannot see an easy way to do this from a plugin, sorry.

I'd still like to understand in more detail why the awareness decider is a bottleneck for you, because maybe there are things we can do to improve its performance. Can you share a test (e.g. an ESAllocationTestCase) that shows a case where it is too slow for you?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.