Temporarily adding nodes or, can you scale down?

Elastic Stack Elasticsearch
1

Hello,

I have not used elasticsearch before, but am considering it for a project.
One question I have is this:

Can you start off with a big number of nodes, in order to speed up the
indexing of legacy data, then scale down for normal operation afterwards?

Thanks a lot,
Mohamed.

--

2

I was hoping my question would get a quick RTFM reply... Maybe it is not
possible to scale down? Or am I thinking about my problem in the wrong way?

Thank you so much,
Mohamed.

On Sunday, September 23, 2012 11:08:29 AM UTC-4, Mohamed Lrhazi wrote:

Hello,

I have not used elasticsearch before, but am considering it for a project.
One question I have is this:

Can you start off with a big number of nodes, in order to speed up the
indexing of legacy data, then scale down for normal operation afterwards?

Thanks a lot,
Mohamed.

--

3

We are running on AWS and index our data on the fastest nodes we can get,
then go back and reduce the nodes to smaller/slower instances to save
money. We don't scale the number of nodes, just the speed/cost of the
nodes. If you're using your own hardware, that would be tougher to do.

The other shoe may be to look at how to index data more efficiently so that
you don't require so many nodes.

  • Craig

On Mon, Sep 24, 2012 at 1:37 PM, Mohamed Lrhazi ml623@georgetown.eduwrote:

I was hoping my question would get a quick RTFM reply... Maybe it is not
possible to scale down? Or am I thinking about my problem in the wrong way?

Thank you so much,
Mohamed.

On Sunday, September 23, 2012 11:08:29 AM UTC-4, Mohamed Lrhazi wrote:

Hello,

I have not used elasticsearch before, but am considering it for a
project. One question I have is this:

Can you start off with a big number of nodes, in order to speed up the
indexing of legacy data, then scale down for normal operation afterwards?

Thanks a lot,
Mohamed.

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

4

Thanks Craig... Are you not lowering the number of nodes because that
is difficult/impossible, or because it's just trivial on AWS?
and thanks for the other shoe idea.

I don't know ES just yet... and I am basically asking: If I use say 10
nodes, I assume my data would end up being distributed more or less evenly
of all 10... Is there a process in ES by which one can move "everything"
(data/index) off of one node, so as to remove it afterwards? even a manual
process?

Thanks a lot,
Mohamed.

On Monday, September 24, 2012 3:43:51 PM UTC-4, Craig Brown wrote:

We are running on AWS and index our data on the fastest nodes we can get,
then go back and reduce the nodes to smaller/slower instances to save
money. We don't scale the number of nodes, just the speed/cost of the
nodes. If you're using your own hardware, that would be tougher to do.

The other shoe may be to look at how to index data more efficiently so
that you don't require so many nodes.

  • Craig

On Mon, Sep 24, 2012 at 1:37 PM, Mohamed Lrhazi <ml...@georgetown.edu<javascript:>

wrote:

I was hoping my question would get a quick RTFM reply... Maybe it is not
possible to scale down? Or am I thinking about my problem in the wrong way?

Thank you so much,
Mohamed.

On Sunday, September 23, 2012 11:08:29 AM UTC-4, Mohamed Lrhazi wrote:

Hello,

I have not used elasticsearch before, but am considering it for a
project. One question I have is this:

Can you start off with a big number of nodes, in order to speed up the
indexing of legacy data, then scale down for normal operation afterwards?

Thanks a lot,
Mohamed.

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

5

I really haven't played around with reducing the number of nodes in the
cluster. I'm not sure of a way to really do that right now, though someone
else may have a better idea. I know there is an upcoming release where you
can manually move shards to particular nodes. The cluster will try to
rebalance the remaining shards across the cluster, but that can be turned
off.

I assume that with the new release you could turn off rebalancing, manually
specify the new shard placements, remove the nodes/nodes, then turn on
rebalancing and have the cluster get back to green status, but I don't know
for sure. Shay would probably be the best one to answer that. I'm not sure
if that release will be 0.19.10 or 0.20.1.

  • Craig

On Mon, Sep 24, 2012 at 2:04 PM, Mohamed Lrhazi ml623@georgetown.eduwrote:

Thanks Craig... Are you not lowering the number of nodes because that
is difficult/impossible, or because it's just trivial on AWS?
and thanks for the other shoe idea.

I don't know ES just yet... and I am basically asking: If I use say 10
nodes, I assume my data would end up being distributed more or less evenly
of all 10... Is there a process in ES by which one can move "everything"
(data/index) off of one node, so as to remove it afterwards? even a manual
process?

Thanks a lot,
Mohamed.

On Monday, September 24, 2012 3:43:51 PM UTC-4, Craig Brown wrote:

We are running on AWS and index our data on the fastest nodes we can get,
then go back and reduce the nodes to smaller/slower instances to save
money. We don't scale the number of nodes, just the speed/cost of the
nodes. If you're using your own hardware, that would be tougher to do.

The other shoe may be to look at how to index data more efficiently so
that you don't require so many nodes.

  • Craig

On Mon, Sep 24, 2012 at 1:37 PM, Mohamed Lrhazi ml...@georgetown.eduwrote:

I was hoping my question would get a quick RTFM reply... Maybe it is not
possible to scale down? Or am I thinking about my problem in the wrong way?

Thank you so much,
Mohamed.

On Sunday, September 23, 2012 11:08:29 AM UTC-4, Mohamed Lrhazi wrote:

Hello,

I have not used elasticsearch before, but am considering it for a
project. One question I have is this:

Can you start off with a big number of nodes, in order to speed up the
indexing of legacy data, then scale down for normal operation afterwards?

Thanks a lot,
Mohamed.

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

6

Very useful info for me, I'll read about the newer releases too.

Thanks a lot,
Mohamed,

On Monday, September 24, 2012 4:13:11 PM UTC-4, Craig Brown wrote:

I really haven't played around with reducing the number of nodes in the
cluster. I'm not sure of a way to really do that right now, though someone
else may have a better idea. I know there is an upcoming release where you
can manually move shards to particular nodes. The cluster will try to
rebalance the remaining shards across the cluster, but that can be turned
off.

I assume that with the new release you could turn off rebalancing,
manually specify the new shard placements, remove the nodes/nodes, then
turn on rebalancing and have the cluster get back to green status, but I
don't know for sure. Shay would probably be the best one to answer that.
I'm not sure if that release will be 0.19.10 or 0.20.1.

  • Craig

On Mon, Sep 24, 2012 at 2:04 PM, Mohamed Lrhazi <ml...@georgetown.edu<javascript:>

wrote:

Thanks Craig... Are you not lowering the number of nodes because that
is difficult/impossible, or because it's just trivial on AWS?
and thanks for the other shoe idea.

I don't know ES just yet... and I am basically asking: If I use say 10
nodes, I assume my data would end up being distributed more or less evenly
of all 10... Is there a process in ES by which one can move "everything"
(data/index) off of one node, so as to remove it afterwards? even a manual
process?

Thanks a lot,
Mohamed.

On Monday, September 24, 2012 3:43:51 PM UTC-4, Craig Brown wrote:

We are running on AWS and index our data on the fastest nodes we can
get, then go back and reduce the nodes to smaller/slower instances to save
money. We don't scale the number of nodes, just the speed/cost of the
nodes. If you're using your own hardware, that would be tougher to do.

The other shoe may be to look at how to index data more efficiently so
that you don't require so many nodes.

  • Craig

On Mon, Sep 24, 2012 at 1:37 PM, Mohamed Lrhazi ml...@georgetown.eduwrote:

I was hoping my question would get a quick RTFM reply... Maybe it is
not possible to scale down? Or am I thinking about my problem in the wrong
way?

Thank you so much,
Mohamed.

On Sunday, September 23, 2012 11:08:29 AM UTC-4, Mohamed Lrhazi wrote:

Hello,

I have not used elasticsearch before, but am considering it for a
project. One question I have is this:

Can you start off with a big number of nodes, in order to speed up the
indexing of legacy data, then scale down for normal operation afterwards?

Thanks a lot,
Mohamed.

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

7

this page tells you how to migrate all the shards off a node
for decommissioning:

the example looks like this:

curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.exclude._ip" : "10.0.0.1"
}
}'

On Mon, Sep 24, 2012 at 1:38 PM, Mohamed Lrhazi ml623@georgetown.eduwrote:

Very useful info for me, I'll read about the newer releases too.

Thanks a lot,
Mohamed,

On Monday, September 24, 2012 4:13:11 PM UTC-4, Craig Brown wrote:

I really haven't played around with reducing the number of nodes in the
cluster. I'm not sure of a way to really do that right now, though someone
else may have a better idea. I know there is an upcoming release where you
can manually move shards to particular nodes. The cluster will try to
rebalance the remaining shards across the cluster, but that can be turned
off.

I assume that with the new release you could turn off rebalancing,
manually specify the new shard placements, remove the nodes/nodes, then
turn on rebalancing and have the cluster get back to green status, but I
don't know for sure. Shay would probably be the best one to answer that.
I'm not sure if that release will be 0.19.10 or 0.20.1.

  • Craig

On Mon, Sep 24, 2012 at 2:04 PM, Mohamed Lrhazi ml...@georgetown.eduwrote:

Thanks Craig... Are you not lowering the number of nodes because that
is difficult/impossible, or because it's just trivial on AWS?
and thanks for the other shoe idea.

I don't know ES just yet... and I am basically asking: If I use say 10
nodes, I assume my data would end up being distributed more or less evenly
of all 10... Is there a process in ES by which one can move "everything"
(data/index) off of one node, so as to remove it afterwards? even a manual
process?

Thanks a lot,
Mohamed.

On Monday, September 24, 2012 3:43:51 PM UTC-4, Craig Brown wrote:

We are running on AWS and index our data on the fastest nodes we can
get, then go back and reduce the nodes to smaller/slower instances to save
money. We don't scale the number of nodes, just the speed/cost of the
nodes. If you're using your own hardware, that would be tougher to do.

The other shoe may be to look at how to index data more efficiently so
that you don't require so many nodes.

  • Craig

On Mon, Sep 24, 2012 at 1:37 PM, Mohamed Lrhazi ml...@georgetown.eduwrote:

I was hoping my question would get a quick RTFM reply... Maybe it is
not possible to scale down? Or am I thinking about my problem in the wrong
way?

Thank you so much,
Mohamed.

On Sunday, September 23, 2012 11:08:29 AM UTC-4, Mohamed Lrhazi wrote:

Hello,

I have not used elasticsearch before, but am considering it for a
project. One question I have is this:

Can you start off with a big number of nodes, in order to speed up
the indexing of legacy data, then scale down for normal operation
afterwards?

Thanks a lot,
Mohamed.

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

--

8

Ya, forgot about that. You can add the nodes you want to remove to the
exclude list and ES will take care of the rest.

  • Craig

On Mon, Sep 24, 2012 at 3:30 PM, David Williams williams.david@gmail.comwrote:

this page tells you how to migrate all the shards off a node
for decommissioning:
Elasticsearch Platform — Find real-time answers at scale | Elastic

the example looks like this:

curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.exclude._ip" : "10.0.0.1"
}
}'

On Mon, Sep 24, 2012 at 1:38 PM, Mohamed Lrhazi ml623@georgetown.eduwrote:

Very useful info for me, I'll read about the newer releases too.

Thanks a lot,
Mohamed,

On Monday, September 24, 2012 4:13:11 PM UTC-4, Craig Brown wrote:

I really haven't played around with reducing the number of nodes in the
cluster. I'm not sure of a way to really do that right now, though someone
else may have a better idea. I know there is an upcoming release where you
can manually move shards to particular nodes. The cluster will try to
rebalance the remaining shards across the cluster, but that can be turned
off.

I assume that with the new release you could turn off rebalancing,
manually specify the new shard placements, remove the nodes/nodes, then
turn on rebalancing and have the cluster get back to green status, but I
don't know for sure. Shay would probably be the best one to answer that.
I'm not sure if that release will be 0.19.10 or 0.20.1.

  • Craig

On Mon, Sep 24, 2012 at 2:04 PM, Mohamed Lrhazi ml...@georgetown.eduwrote:

Thanks Craig... Are you not lowering the number of nodes because that
is difficult/impossible, or because it's just trivial on AWS?
and thanks for the other shoe idea.

I don't know ES just yet... and I am basically asking: If I use say 10
nodes, I assume my data would end up being distributed more or less evenly
of all 10... Is there a process in ES by which one can move "everything"
(data/index) off of one node, so as to remove it afterwards? even a manual
process?

Thanks a lot,
Mohamed.

On Monday, September 24, 2012 3:43:51 PM UTC-4, Craig Brown wrote:

We are running on AWS and index our data on the fastest nodes we can
get, then go back and reduce the nodes to smaller/slower instances to save
money. We don't scale the number of nodes, just the speed/cost of the
nodes. If you're using your own hardware, that would be tougher to do.

The other shoe may be to look at how to index data more efficiently so
that you don't require so many nodes.

  • Craig

On Mon, Sep 24, 2012 at 1:37 PM, Mohamed Lrhazi ml...@georgetown.eduwrote:

I was hoping my question would get a quick RTFM reply... Maybe it is
not possible to scale down? Or am I thinking about my problem in the wrong
way?

Thank you so much,
Mohamed.

On Sunday, September 23, 2012 11:08:29 AM UTC-4, Mohamed Lrhazi wrote:

Hello,

I have not used elasticsearch before, but am considering it for a
project. One question I have is this:

Can you start off with a big number of nodes, in order to speed up
the indexing of legacy data, then scale down for normal operation
afterwards?

Thanks a lot,
Mohamed.

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

9

Alright... one curl command to downsize my cluster... How cool is that?

Thanks a lot guys,
Mohamed.

On Monday, September 24, 2012 5:53:17 PM UTC-4, Craig Brown wrote:

Ya, forgot about that. You can add the nodes you want to remove to the
exclude list and ES will take care of the rest.

  • Craig

On Mon, Sep 24, 2012 at 3:30 PM, David Williams <william...@gmail.com<javascript:>

wrote:

this page tells you how to migrate all the shards off a node
for decommissioning:
Elasticsearch Platform — Find real-time answers at scale | Elastic

the example looks like this:

curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.exclude._ip" : "10.0.0.1"
}
}'

On Mon, Sep 24, 2012 at 1:38 PM, Mohamed Lrhazi <ml...@georgetown.edu<javascript:>

wrote:

Very useful info for me, I'll read about the newer releases too.

Thanks a lot,
Mohamed,

On Monday, September 24, 2012 4:13:11 PM UTC-4, Craig Brown wrote:

I really haven't played around with reducing the number of nodes in the
cluster. I'm not sure of a way to really do that right now, though someone
else may have a better idea. I know there is an upcoming release where you
can manually move shards to particular nodes. The cluster will try to
rebalance the remaining shards across the cluster, but that can be turned
off.

I assume that with the new release you could turn off rebalancing,
manually specify the new shard placements, remove the nodes/nodes, then
turn on rebalancing and have the cluster get back to green status, but I
don't know for sure. Shay would probably be the best one to answer that.
I'm not sure if that release will be 0.19.10 or 0.20.1.

  • Craig

On Mon, Sep 24, 2012 at 2:04 PM, Mohamed Lrhazi ml...@georgetown.eduwrote:

Thanks Craig... Are you not lowering the number of nodes because that
is difficult/impossible, or because it's just trivial on AWS?
and thanks for the other shoe idea.

I don't know ES just yet... and I am basically asking: If I use say 10
nodes, I assume my data would end up being distributed more or less evenly
of all 10... Is there a process in ES by which one can move "everything"
(data/index) off of one node, so as to remove it afterwards? even a manual
process?

Thanks a lot,
Mohamed.

On Monday, September 24, 2012 3:43:51 PM UTC-4, Craig Brown wrote:

We are running on AWS and index our data on the fastest nodes we can
get, then go back and reduce the nodes to smaller/slower instances to save
money. We don't scale the number of nodes, just the speed/cost of the
nodes. If you're using your own hardware, that would be tougher to do.

The other shoe may be to look at how to index data more efficiently
so that you don't require so many nodes.

  • Craig

On Mon, Sep 24, 2012 at 1:37 PM, Mohamed Lrhazi <ml...@georgetown.edu

wrote:

I was hoping my question would get a quick RTFM reply... Maybe it is
not possible to scale down? Or am I thinking about my problem in the wrong
way?

Thank you so much,
Mohamed.

On Sunday, September 23, 2012 11:08:29 AM UTC-4, Mohamed Lrhazi
wrote:

Hello,

I have not used elasticsearch before, but am considering it for a
project. One question I have is this:

Can you start off with a big number of nodes, in order to speed up
the indexing of legacy data, then scale down for normal operation
afterwards?

Thanks a lot,
Mohamed.

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--

--

--
…
CRAIG BROWN
youwho, Inc.

www.youwho.com

T: 801.855. 0921
M: 801.913. 0939

--