Start elasticsearch that used to be in a cluster as a single-node or in a different cluster

Elastic Stack Elasticsearch
1

Hey Everyone,

Due to some stuff that happened, I have an Elasticsearch node with a lot of data, that isn't up to date with the cluster.

What I need to do is somehow start this node as a separate cluster, without it needing the voting results of two other nodes, and without it joining the existing cluster.

How would I go about doing this?

Cheers,
Luka

2

One more thing, if it's not possible to transfer it as a single-node. Is it possible to create new voting-only masters and join them to the new cluster so it starts up?

Cheers,
Luka

3

Which version of Elasticsearch are you using?

Is the node master eligible?

How did you end up in this situation? Why can the node not rejoin the cluster?

4

Which version of Elasticsearch are you using?

7.15.2

Is the node master eligible?

yes

How did you end up in this situation? Why can the node not rejoin the cluster?

The FS got corrupted so we spent a few days recovering it... A lot of stuff happened, but long story short - if he joins the cluster he will delete the indices because they are no longer present on the current cluster (they were deleted).

5

I suspect you will need to use the elasticsearch-node tool, but note that this comes with warnings and is unsafe. I will not be able to help with this as I have fortunately not had to use it, but maybe someone else can help if you have issues or questions around the docs.

6

Thanks @Christian_Dahlqvist, so far so good.
The unsafe-bootstrap bootstrap option allowed me to start the node with a different cluster name and the data is safe, and in tact.

Now it's just about getting the data synced.

Cheers,
Luka

7

Since that node is now it's own cluster. Could you use logstash to move the data between the two clusters? Might not be quick, but you'd have good visibility of it working as the process runs.

8

I would recommend either moving the data through snapshot/restore or do a remote reindexing.

9

We did a lot of testing on a massive cluster we have and Logstash is only good for a small amount of data. Same goes for a remote reindex (since slicing is not supported).

No matter how much we tuned a single logstash couldn't go over 60k/s ingesting and a remote reindex capped at about 30-35k/s.

Another bad thing with Logstash is it has no state when using elasticsearch as input and output. If it restarts or crashes or smth and you have to restart it, it will re-read all the indexes (assuming you give it a wildcard). If going index by index you have to monitor it constantly which is also not ideal.

Best bet in case of smaller data, imo is remote reindex as it can be monitored via task API.

In big clusters, snapshot/restore is by FAR the best bet.

Cheers,
Luka

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.