Hi folks. I'm using Elasticsearch 0.18.7 as backend for Logstash and I've
never had to do any administration before, so I'm quite a new user of this
software.
Problem arose because we're moving any service of ours form 32bit to 64bit
architecture, and I hoped to move Elasticsearch as well just moving the
data around. Problem is that I wasn't able to get Elasticsearch with old
data to work in that way and so our collection of Logstash data is now
"forked" between the new set on the 64bit box, and the old data that we'd
like to get back into the new instance. Yeah, I know it's quite messy...
Any chance this can be accomplished? Thanks for your time
Hi folks. I'm using Elasticsearch 0.18.7 as backend for Logstash and I've
never had to do any administration before, so I'm quite a new user of this
software.
Problem arose because we're moving any service of ours form 32bit to 64bit
architecture, and I hoped to move Elasticsearch as well just moving the
data around. Problem is that I wasn't able to get Elasticsearch with old
data to work in that way and so our collection of Logstash data is now
"forked" between the new set on the 64bit box, and the old data that we'd
like to get back into the new instance. Yeah, I know it's quite messy...
Any chance this can be accomplished? Thanks for your time
You mean you now span a cluster between 32 and 64 bit machines? That
won't be a problem.
Not exactly. ES on the 32bit machine now is not used and I'd like to get
data stored there to the new 64bit machine. In the end there will be
just one ES running on 64, hopefully with old data too.
You have several options. The first is simply copying the data directory
from the 32 bit node to the new node, and then shutdown the old node and
start the new node. The second is to do live migration. For this, you can
start the new 64 node and have it join the cluster with the 32 bit node,
onces it has joined the cluster (you can configure the new node with
unicast discovery pointing to the 32 and 64 bit nodes), you have two
options:
If you already have number_of_replicas set to 1, then you replicas will
be allocated on the new 64bit node. Once the health is green (replicas are
allocated), then you can shutdown the old node and set the number of
replicas to 0. If you initially started with 0 replicas, then you can use
the indices update settings API to increase it to 1 so they will be
replicated on the new node.
Once hte cluster is formed, you can issue a cluster update settings API
and decommission the 32 bit node using cluster exclude allocation filtering
(you can do that based on IP, for example:
You mean you now span a cluster between 32 and 64 bit machines? That
won't be a problem.
Not exactly. ES on the 32bit machine now is not used and I'd like to get
data stored there to the new 64bit machine. In the end there will be
just one ES running on 64, hopefully with old data too.
You have several options. The first is simply copying the data directory
from the 32 bit node to the new node, and then shutdown the old node and
start the new node. The second is to do live migration. For this, you
can start the new 64 node and have it join the cluster with the 32 bit
node, onces it has joined the cluster (you can configure the new node
with unicast discovery pointing to the 32 and 64 bit nodes), you have
two options:
Hi Shay, thank you very much for your support so far! First I'll try the
first way that seems the most straightforward (I can afford a little of
downtime). I'm just puzzled with one thing: what if I want to "merge"
indices still on the old machine into the new one keeping the new data
collected on the new box there? If I just copy the data dir I guess it
will overwrite the new collected data, right?
Yes, it will overwrite it, if you want to merge it, you need to reindex the
old data into the old data. Another option is to simply copy the daily
indices from the old one to the new cluster, since logstash by default
rollover by day.
You have several options. The first is simply copying the data directory
from the 32 bit node to the new node, and then shutdown the old node and
start the new node. The second is to do live migration. For this, you
can start the new 64 node and have it join the cluster with the 32 bit
node, onces it has joined the cluster (you can configure the new node
with unicast discovery pointing to the 32 and 64 bit nodes), you have
two options:
Hi Shay, thank you very much for your support so far! First I'll try the
first way that seems the most straightforward (I can afford a little of
downtime). I'm just puzzled with one thing: what if I want to "merge"
indices still on the old machine into the new one keeping the new data
collected on the new box there? If I just copy the data dir I guess it
will overwrite the new collected data, right?
A small update: I'm trying the "set up a cluster" suggested way, but
seems that using v0.18.7, even adding the two nodes to the same cluster
makes new indices disappear from the new (64bit) box.
A small update: I'm trying the "set up a cluster" suggested way, but
seems that using v0.18.7, even adding the two nodes to the same cluster
makes new indices disappear from the new (64bit) box.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
