What is the best way to transfer data when you change your cluster name?

GordonM · June 28, 2015, 3:04pm

I am working on a cluster with just a single node for development.

I tried changing my cluster name, and copied the entire nodes folder from /var/lib/elasticsearch/(old cluster folder name) to var/lib/elasticsearch/(new cluster folder name) but kibana cannot find the indexes from my old cluster name. And when I had new incoming log data imported into them from logstash, kibana found them as if they were new and did not have any previous records.

What’s going on here?

I eventually did this, which seemed to work, but it seems like there may be issues:

Stop elasticsearch
Rename the elasticsearch cluster in elasticsearch.yml
Go to /var/lib/elasticsearch and preemptively copy the old cluster folder into the new cluster folder using the new cluster’s exact name specified in elasticsearch.yml
Start elasticsearch

Any thoughts here?

warkolm · June 28, 2015, 9:37pm

That's how it works, if you change the cluster name it changes the directory structure it stores the indices in.

GordonM · June 28, 2015, 11:27pm

Any idea on why just copying the nodes folder from one cluster folder to another does not work? I read from others this was successful, but I encountered the difficulties in the OP.

warkolm · June 28, 2015, 11:47pm

I just tried that and it worked. Do you have the full commands you ran, with the output?

GordonM · June 29, 2015, 12:15am

I will have to try again tomorrow. I think it worked for me once, too, but tried it again and was unsuccessful so I thought just copying and pasting might be unreliable. I think the second time I had incoming logs/documents, while the first time I had turned off incoming logs/documents. My guess was that incoming documents were causing elasticsearch to auto-create cluster information that was not replaced when I copied over previous cluster's node folder.

magnusbaeck · June 29, 2015, 5:35am

Well, you definitely don't want to have ES running while you're doing this that could definitely explain why things didn't work. Not sure why you want to copy the data directory—why not just rename it?

GordonM · June 30, 2015, 3:22pm

I think that was it. I tried it again and it worked. I am not sure how I forgot to turn off elasticsearch though. So I guess the thing to remember is turn off elasticsearch and keep it off while editing elasticsearch.yml and copying the cluster's folder, and to not turn it on at intervening steps. (I think that's what must have happened - I recall turning it on to check to make sure the cluster's name was really changed.)