Rsync data folder "nodes/0/indices" from broken node to working node

Elastic Stack Elasticsearch
1

I couldn't find a recent answer for this (but my search skills after 16 hours of 5-to-6-upgrade-work are a little... downscaled). Long story short: I have to move 14 shards from a broken-beyond-repair node, which is booted in rescue mode, to a working node.

Can I simply rsync -a --ignore-existing /var/lib/elasticsearch/nodes/0/indices from the broken node to the working node?

Do I need to stop the working node during the sync?

Thanks in advance!

2

Turns out: yes, you totally can. Well, better to check the index uuid and select just the missing shards, otherwise you wait quite a while to copy 600GB from one shard via a 1GBit connection.

That was exhaustive... but the numbers look good so far, the garbage collection nightmare with 5.6 and extensive-terms-query caching seems to be resolved, hooray.