Possible host awareness when doing shard relocation?

[zjxian (Zhang Junxian)]

We're running a cluster that each host (physical machine) has 2 es nodes running. On each host, 1 node (hot ) uses the SSD as storage while another ( stale ) uses SATA as storage. We are using shard allocation filter to control the index, so that recent indices are kept on hot nodes and older indices are kept on stale nodes. When some hot indices become "old", we change shard allocation filters so that es cluster automatically relocate them from hot to stale.
However, the relocation is quite random. Suppose 3 hosts, H1 to H3. What I hope to see, is that shards on hot_node_H1 is relocate to stale_node_H1, and so on. But usually shards on hot_node_H1 is relocate to stale_node_H3, and cause what I think unnecessary network flows.
So is there any related config to achieve this, so that in my scenario the relocation may be more efficient ?

[s1monw (Simon Willnauer)]

ok lemme ask some questions.

• stale_node_H3 and hot_node_H3 is the same physical machine?
• given my previous questions answer is yes why would you wanna do this can't you just leave it there?
• if you move from any given node X to any given node Y we push stuff over the network you mean it could be more efficient to push it through loopback on the same node, correct?
• if you already prevent the allocation on hot_node_H3 via allocation filters why don't you use these filters to advice the system to move to stale_node_H3?
• also if you have 2 nodes running on the same physical HW you should also set cluster.routing.allocation.same_shard.host: true to tell elasticsearch to not place any replicas on the same physical machine. If you do that the relocation won't work to the same physical HW.

simon

[zjxian (Zhang Junxian)]

@s1monw

ok lemme ask some questions.

• stale_node_H3 and hot_node_H3 is the same physical machine?

Yes. Each host in my cluster has both SSD and SATA disks. Hot node uses SSD as storage, which is relatively small (240GB each node), stale node uses 3x6TB SATA disk (each node).
Currently the cluster uses 3 hosts ( 6 data nodes, 3 hot, 3 stale ), and a dedicated host as master node.
The indices are generated on a monthly basis. The storage spaces on all hot nodes may hold 2~3 month, but we're planning for long term (18~24 month). So we have to move indices from SSD to SATA disks.

• given my previous questions answer is yes why would you wanna do this can't you just leave it there?

As stated above, the SSD storage is not enough for long term.

• if you move from any given node X to any given node Y we push stuff over the network you mean it could be more efficient to push it through loopback on the same node, correct?

Yes. The network between each physical machine in our cluster is relatively poor, just 1Gbps, and is quite problematic to improve. So we're thinking transferring data through loopback may speed things up.

• if you already prevent the allocation on hot_node_H3 via allocation filters why don't you use these filters to advice the system to move to stale_node_H3?

Maybe there is some mis-understanding. Currently we're using allocation filters to advice the system to move all shards that previously on all the hot node, to all the stale nodes.
The problem is, the cluster consists of several hosts. And after using allocation filter (exclude all hot nodes and include all stale nodes), for example the shards previous on hot_node_H3 may be relocated to stale_node_H2, which is the stale node on another host machine. And data is transferred through the 1Gps network.

ES seems quite random when choosing the destination....

• also if you have 2 nodes running on the same physical HW you should also set cluster.routing.allocation.same_shard.host: true to tell elasticsearch to not place any replicas on the same physical machine. If you do that the relocation won't work to the same physical HW.

Replica setting is currently not our concern. The problem is we can't find a way to coordinate “shard relocation" in index relocation process.

simon

Thank you for your reply.