Dumping data from 32bit to 64bit


(Alexander Fortin) #1

Hi folks. I'm using Elasticsearch 0.18.7 as backend for Logstash and I've
never had to do any administration before, so I'm quite a new user of this
software.

Problem arose because we're moving any service of ours form 32bit to 64bit
architecture, and I hoped to move Elasticsearch as well just moving the
data around. Problem is that I wasn't able to get Elasticsearch with old
data to work in that way and so our collection of Logstash data is now
"forked" between the new set on the 64bit box, and the old data that we'd
like to get back into the new instance. Yeah, I know it's quite messy...

Any chance this can be accomplished? Thanks for your time


(Shay Banon) #2

You mean you now span a cluster between 32 and 64 bit machines? That won't
be a problem.

On Tue, Apr 10, 2012 at 1:30 PM, Alexander Fortin <
alexander.fortin@gmail.com> wrote:

Hi folks. I'm using Elasticsearch 0.18.7 as backend for Logstash and I've
never had to do any administration before, so I'm quite a new user of this
software.

Problem arose because we're moving any service of ours form 32bit to 64bit
architecture, and I hoped to move Elasticsearch as well just moving the
data around. Problem is that I wasn't able to get Elasticsearch with old
data to work in that way and so our collection of Logstash data is now
"forked" between the new set on the 64bit box, and the old data that we'd
like to get back into the new instance. Yeah, I know it's quite messy...

Any chance this can be accomplished? Thanks for your time


(Alexander Fortin) #3

On 4/11/12 11:45 AM, Shay Banon wrote:

You mean you now span a cluster between 32 and 64 bit machines? That
won't be a problem.

Not exactly. ES on the 32bit machine now is not used and I'd like to get
data stored there to the new 64bit machine. In the end there will be
just one ES running on 64, hopefully with old data too.

--
Alexander Fortin


(Shay Banon) #4

You have several options. The first is simply copying the data directory
from the 32 bit node to the new node, and then shutdown the old node and
start the new node. The second is to do live migration. For this, you can
start the new 64 node and have it join the cluster with the 32 bit node,
onces it has joined the cluster (you can configure the new node with
unicast discovery pointing to the 32 and 64 bit nodes), you have two
options:

  1. If you already have number_of_replicas set to 1, then you replicas will
    be allocated on the new 64bit node. Once the health is green (replicas are
    allocated), then you can shutdown the old node and set the number of
    replicas to 0. If you initially started with 0 replicas, then you can use
    the indices update settings API to increase it to 1 so they will be
    replicated on the new node.

  2. Once hte cluster is formed, you can issue a cluster update settings API
    and decommission the 32 bit node using cluster exclude allocation filtering
    (you can do that based on IP, for example:

curl -XPUT localhost:9200/_cluster/_settings -d '{
"transient" : {
"cluster.routing.allocation.exclude._ip" : "10.0.0.1"
}
}'

And then, wait for green status to have all teh shards migrate to the new
node, and then shutdown the node.

On Wed, Apr 11, 2012 at 3:50 PM, Alexander Fortin <
alexander.fortin@gmail.com> wrote:

On 4/11/12 11:45 AM, Shay Banon wrote:

You mean you now span a cluster between 32 and 64 bit machines? That
won't be a problem.

Not exactly. ES on the 32bit machine now is not used and I'd like to get
data stored there to the new 64bit machine. In the end there will be
just one ES running on 64, hopefully with old data too.

--
Alexander Fortin
http://about.me/alexanderfortin/


(Alexander Fortin) #5

On 4/11/12 8:43 PM, Shay Banon wrote:

You have several options. The first is simply copying the data directory
from the 32 bit node to the new node, and then shutdown the old node and
start the new node. The second is to do live migration. For this, you
can start the new 64 node and have it join the cluster with the 32 bit
node, onces it has joined the cluster (you can configure the new node
with unicast discovery pointing to the 32 and 64 bit nodes), you have
two options:

Hi Shay, thank you very much for your support so far! First I'll try the
first way that seems the most straightforward (I can afford a little of
downtime). I'm just puzzled with one thing: what if I want to "merge"
indices still on the old machine into the new one keeping the new data
collected on the new box there? If I just copy the data dir I guess it
will overwrite the new collected data, right?

--
Alexander Fortin


(Shay Banon) #6

Yes, it will overwrite it, if you want to merge it, you need to reindex the
old data into the old data. Another option is to simply copy the daily
indices from the old one to the new cluster, since logstash by default
rollover by day.

On Thu, Apr 12, 2012 at 7:02 PM, Alexander Fortin <
alexander.fortin@gmail.com> wrote:

On 4/11/12 8:43 PM, Shay Banon wrote:

You have several options. The first is simply copying the data directory
from the 32 bit node to the new node, and then shutdown the old node and
start the new node. The second is to do live migration. For this, you
can start the new 64 node and have it join the cluster with the 32 bit
node, onces it has joined the cluster (you can configure the new node
with unicast discovery pointing to the 32 and 64 bit nodes), you have
two options:

Hi Shay, thank you very much for your support so far! First I'll try the
first way that seems the most straightforward (I can afford a little of
downtime). I'm just puzzled with one thing: what if I want to "merge"
indices still on the old machine into the new one keeping the new data
collected on the new box there? If I just copy the data dir I guess it
will overwrite the new collected data, right?

--
Alexander Fortin
http://about.me/alexanderfortin/


(Alexander Fortin) #7

On 4/13/12 2:26 PM, Shay Banon wrote:

Yes, it will overwrite it, if you want to merge it, you need to reindex
the old data into the old data.
^^^
I guess you meant into the new data :wink:

Another option is to simply copy the
daily indices from the old one to the new cluster, since logstash by
default rollover by day.

Hum, actually that was the first thing I tried but didn't seem to work
to me, I'll give it another try maybe I missed something in the process.

--
Alexander Fortin


(Alexander Fortin) #8

On 4/15/12 2:56 PM, Alexander Fortin wrote:

Hum, actually that was the first thing I tried but didn't seem to work
to me, I'll give it another try maybe I missed something in the process.

Actually, just copying across the indices, something weird happens:
seems that ES removes any index that has been copied:
http://pastebin.com/hJVKkbgW

--
Alexander Fortin


(Alexander Fortin) #9

On 4/15/12 4:17 PM, Alexander Fortin wrote:

On 4/15/12 2:56 PM, Alexander Fortin wrote:

Hum, actually that was the first thing I tried but didn't seem to work
to me, I'll give it another try maybe I missed something in the process.
Actually, just copying across the indices, something weird happens:
seems that ES removes any index that has been copied:
http://pastebin.com/hJVKkbgW

A small update: I'm trying the "set up a cluster" suggested way, but
seems that using v0.18.7, even adding the two nodes to the same cluster
makes new indices disappear from the new (64bit) box.

--
Alexander Fortin


(Shay Banon) #10

You need to have 0.19 for that.

On Wed, Apr 18, 2012 at 12:54 PM, Alexander Fortin <
alexander.fortin@gmail.com> wrote:

On 4/15/12 4:17 PM, Alexander Fortin wrote:

On 4/15/12 2:56 PM, Alexander Fortin wrote:

Hum, actually that was the first thing I tried but didn't seem to work
to me, I'll give it another try maybe I missed something in the
process.
Actually, just copying across the indices, something weird happens:
seems that ES removes any index that has been copied:
http://pastebin.com/hJVKkbgW

A small update: I'm trying the "set up a cluster" suggested way, but
seems that using v0.18.7, even adding the two nodes to the same cluster
makes new indices disappear from the new (64bit) box.

--
Alexander Fortin
http://about.me/alexanderfortin/


(system) #11