Allocate specific shards on specific nodes

Elastic Stack Elasticsearch
1

Hi,
given that I have a cluster with for example 8 nodes
elastic1 -> elastic8
and I create an index with 8 shards, is there a way to enforce the following from the start:
shard 1 should be on elastic1,
shard 2 on elastic2,
shard 3 on elastic3
..
shard 8 on elastic8.

I know you can use the _reroute api afterwards based on how elastic routed the shards initially but I would like to have something where I can tell elasticsearch to do this from the start.

I need this because I have a plugin that keeps a state in memory for certain documents,
and I need to know for sure if a new index is made on what node a certain shard is on.(needs to be same for new indexes)

2

You cannot do this automatically, and you don't want to.

This seems extremely dangerous.

3

There are ways to influence shard allocation, for instance based on racks or how beefy machines are. That said I am not sure these apply to your case. Also, I don't follow why you would need to do that. You mentioned that you have a plugin that needs to know and maybe that is the root of the problem. Maybe you can explain a bit more about the plugin and the reason behind this requirement though.

4

I still think there are more advanced use cases where you want to have full control over allocations.

For example if you start with 8 equal servers with x amount of (smaller) indices having each a different amount of shards. If you would then create a new larger index that gets searched a lot with 8 shards, it could be that 2 shards are assigned to one node and that one node would bring down the search performance. Because you're not using a well balanced index over all nodes.

I think it's a limitation of the current index create API if you're able to do the allocation with the reroute api immediately afterwards. Either you don't expose the _reroute api or have similar functionality available from the start in the create api.

The plugin is my own plugin, it tracks usage of certain documents that I need in ordering (score changes based on usage). It needs to be fast and expire after a while. I can't use an attribute in the index as this would need disk acces and recreation of the document each time the attribute gets updated. I use a fixed guava java cache, that expires after x amount of time so it will never expose a threat to the server memory.

Anyways I've managed to get it working using the _reroute api now.

5

Did you have a look at shard allocation filtering? Sounds like it would help in your case, and would be much less manual that using the reroute api.

Cheers
Luca