What is the best way to transfer data when you change your cluster name?

Elastic Stack Elasticsearch
1

What is the best way to transfer data when you change your cluster name?

I am working on a cluster with just a single node for development.

I tried changing my cluster name, and copied the entire nodes folder from /var/lib/elasticsearch/(old cluster folder name) to var/lib/elasticsearch/(new cluster folder name) but kibana cannot find the indexes from my old cluster name. And when I had new incoming log data imported into them from logstash, kibana found them as if they were new and did not have any previous records.

What’s going on here?

I eventually did this, which seemed to work, but it seems like there may be issues:

  1. Stop elasticsearch
  2. Rename the elasticsearch cluster in elasticsearch.yml
  3. Go to /var/lib/elasticsearch and preemptively copy the old cluster folder into the new cluster folder using the new cluster’s exact name specified in elasticsearch.yml
  4. Start elasticsearch

Any thoughts here?

2

That's how it works, if you change the cluster name it changes the directory structure it stores the indices in.

3

Any idea on why just copying the nodes folder from one cluster folder to another does not work? I read from others this was successful, but I encountered the difficulties in the OP.

4

I just tried that and it worked. Do you have the full commands you ran, with the output?

5

I will have to try again tomorrow. I think it worked for me once, too, but tried it again and was unsuccessful so I thought just copying and pasting might be unreliable. I think the second time I had incoming logs/documents, while the first time I had turned off incoming logs/documents. My guess was that incoming documents were causing elasticsearch to auto-create cluster information that was not replaced when I copied over previous cluster's node folder.

6

Well, you definitely don't want to have ES running while you're doing this that could definitely explain why things didn't work. Not sure why you want to copy the data directory—why not just rename it?

7

I think that was it. I tried it again and it worked. I am not sure how I forgot to turn off elasticsearch though. So I guess the thing to remember is turn off elasticsearch and keep it off while editing elasticsearch.yml and copying the cluster's folder, and to not turn it on at intervening steps. (I think that's what must have happened - I recall turning it on to check to make sure the cluster's name was really changed.)