Moving whole old index

Elastic Stack Elasticsearch
1

Hi everyone!

I got a 4 nodes cluster, 3 of them have 100Gb SSD and a other have a 4 Tb
HDD.
Each indexes has 6 shards.
Since every daily index is using around 50Gb I cannot store more than 6
days.

I would like to store old indexes to the 4 Tb HDD.

Is there a good way to move the old shards/indexes automatically, like with
a cron script?

Thanks.

Cheers,
Krager

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

I'd love to know this as well, as I have servers with both SSD and HDD
arrays where i'd like to store the most recent days data on SSD, then
migrate to HDDs.

Note: I have not tested this yet, but this was my 'plan of attack' to see
if I could do this:

I thought to use rerouting:
Elasticsearch Platform — Find real-time answers at scale | Elastic -
which looks like it can be easily enough scripted to force index/shard
movement to a slower array. But it looks like the cluster will re-balance
after any forced re-route.

So, I was going to try using index shard allocation config: (
Elasticsearch Platform — Find real-time answers at scale | Elastic)
to exclude my 'slow' node, and then only force it on manual re-route above.

-Zachary

On Monday, October 21, 2013 9:23:22 AM UTC-5, Krager wrote:

Hi everyone!

I got a 4 nodes cluster, 3 of them have 100Gb SSD and a other have a 4 Tb
HDD.
Each indexes has 6 shards.
Since every daily index is using around 50Gb I cannot store more than 6
days.

I would like to store old indexes to the 4 Tb HDD.

Is there a good way to move the old shards/indexes automatically, like
with a cron script?

Thanks.

Cheers,
Krager

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

That's the direction I was going to.
But I would have excluded SSD nodes and required the HDD when index gets
too old, then doing a reroute so that it won't be rebalanced after. But
reroute could be a pain as you've got to specify each shard and from which
node it is coming from.
I haven't tested this still, and I wonder if there is a way to move the
whole index or moving each shards wherever they're coming from.

--Krager

Le lundi 21 octobre 2013 16:46:45 UTC+2, Zachary Lammers a écrit :

I'd love to know this as well, as I have servers with both SSD and HDD
arrays where i'd like to store the most recent days data on SSD, then
migrate to HDDs.

Note: I have not tested this yet, but this was my 'plan of attack' to see
if I could do this:

I thought to use rerouting:
Elasticsearch Platform — Find real-time answers at scale | Elastic -
which looks like it can be easily enough scripted to force index/shard
movement to a slower array. But it looks like the cluster will re-balance
after any forced re-route.

So, I was going to try using index shard allocation config: (
Elasticsearch Platform — Find real-time answers at scale | Elastic)
to exclude my 'slow' node, and then only force it on manual re-route above.

-Zachary

On Monday, October 21, 2013 9:23:22 AM UTC-5, Krager wrote:

Hi everyone!

I got a 4 nodes cluster, 3 of them have 100Gb SSD and a other have a 4 Tb
HDD.
Each indexes has 6 shards.
Since every daily index is using around 50Gb I cannot store more than 6
days.

I would like to store old indexes to the 4 Tb HDD.

Is there a good way to move the old shards/indexes automatically, like
with a cron script?

Thanks.

Cheers,
Krager

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Ok so, it seems there is no need, to reroute.

I just did an exclude for SSD and include on HDD and the whole index moved
automatically (on live cluster).

The index doesn't seems to be rerouted so far it looks just good for now.
Here is the command:
curl -XPUT localhost:9200/oldindex/_settings -d '{
"index.routing.allocation.exclude._ip" : "X.X.X.X,Y.Y.Y.Y",
"index.routing.allocation.include._ip" : "Z.Z.Z.Z"
}'

Of course, oldindex is a random index name and X.X.X.X stands for the IP
address of the node

Cheers,

Krager

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

I was able to do my test as well, almost same command as below, but I did
it with tags -- on my nodes, i have a "node.tag: [current|archive]", and my
template excludes my 'archive' tag in the settings (
"index.routing.allocation.exclude.tag" : "archive" ) Then, to move my
index, I ran the very similar command below. I couldn't use IPs like you
did, as I have large SSD arrays and HDD arrays on the same server, so I run
a 'current store' ES node and an 'archive' ES node at the same time.

curl -XPUT localhost:9200//_settings -d '{
"index.routing.allocation.exclude.tag" : "current",
"index.routing.allocation.include.tag" : "archive"
}'

On Tuesday, October 22, 2013 10:05:44 AM UTC-5, Krager wrote:

Ok so, it seems there is no need, to reroute.

I just did an exclude for SSD and include on HDD and the whole index moved
automatically (on live cluster).

The index doesn't seems to be rerouted so far it looks just good for now.
Here is the command:
curl -XPUT localhost:9200/oldindex/_settings -d '{
"index.routing.allocation.exclude._ip" : "X.X.X.X,Y.Y.Y.Y",
"index.routing.allocation.include._ip" : "Z.Z.Z.Z"
}'

Of course, oldindex is a random index name and X.X.X.X stands for the IP
address of the node

Cheers,

Krager

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

1 Like