Shard allocation awareness (forced too) and replica allocation

Elastic Stack Elasticsearch
1

Hi everyone,

A bit of background:

  • Cluster of six nodes, three tagged as being "indexers" and three tagged
    as being "searchers"
  • One index, three shards, replication of one.

I'd like to know if it's possible to have a replication factor of three so
all the "searchers" have a replica of all the shards. I've tried it and
both the searchers and indexers ended up with two shards each and when I
tried to move the replica shards from the "indexers" to the "searchers",
elastic returned me the following:

{"error":"RemoteTransportException[[node0005][inet[/ip_address:9301]][cluster/reroute]];
nested: ElasticSearchIllegalArgumentException[[move_allocation] can't move
[hopper][1], from
[node0006][KylcDE8ZTL6KfETSMecxzQ][inet[node0006/ip_address:9301]]{role=index,
master=true}, to
[node0003][PjcZhF73SxmS8G-wQkSL0A][inet[node0003/ip_address:9301]]{role=search,
master=true}, since its not allowed, reason:
[YES()][YES()][YES()][YES()][YES()][NO()][YES()]]; ","status":400}

Any ideas?

Thanks!

Sebastien

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

By the way, I'm using 0.90.0 :slight_smile:

On Friday, May 24, 2013 5:01:10 PM UTC-4, Sebastien Coutu wrote:

Hi everyone,

A bit of background:

  • Cluster of six nodes, three tagged as being "indexers" and three tagged
    as being "searchers"
  • One index, three shards, replication of one.

I'd like to know if it's possible to have a replication factor of three so
all the "searchers" have a replica of all the shards. I've tried it and
both the searchers and indexers ended up with two shards each and when I
tried to move the replica shards from the "indexers" to the "searchers",
elastic returned me the following:

{"error":"RemoteTransportException[[node0005][inet[/ip_address:9301]][cluster/reroute]];
nested: ElasticSearchIllegalArgumentException[[move_allocation] can't move
[hopper][1], from
[node0006][KylcDE8ZTL6KfETSMecxzQ][inet[node0006/ip_address:9301]]{role=index,
master=true}, to
[node0003][PjcZhF73SxmS8G-wQkSL0A][inet[node0003/ip_address:9301]]{role=search,
master=true}, since its not allowed, reason:
[YES()][YES()][YES()][YES()][YES()][NO()][YES()]]; ","status":400}

Any ideas?

Thanks!

Sebastien

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Are you saying that the searchers have no shards? Both primaries and
replicas are on the indexers?

Please post your config (and any index settings)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

some factors in the allocation decision prevent a move of a shard here, can
you tell what kind of setting you have aside of the defaults related to
allocation and the number of machines ie. your setup?

simon

On Friday, May 24, 2013 11:01:33 PM UTC+2, Sebastien Coutu wrote:

By the way, I'm using 0.90.0 :slight_smile:

On Friday, May 24, 2013 5:01:10 PM UTC-4, Sebastien Coutu wrote:

Hi everyone,

A bit of background:

  • Cluster of six nodes, three tagged as being "indexers" and three tagged
    as being "searchers"
  • One index, three shards, replication of one.

I'd like to know if it's possible to have a replication factor of three
so all the "searchers" have a replica of all the shards. I've tried it and
both the searchers and indexers ended up with two shards each and when I
tried to move the replica shards from the "indexers" to the "searchers",
elastic returned me the following:

{"error":"RemoteTransportException[[node0005][inet[/ip_address:9301]][cluster/reroute]];
nested: ElasticSearchIllegalArgumentException[[move_allocation] can't move
[hopper][1], from
[node0006][KylcDE8ZTL6KfETSMecxzQ][inet[node0006/ip_address:9301]]{role=index,
master=true}, to
[node0003][PjcZhF73SxmS8G-wQkSL0A][inet[node0003/ip_address:9301]]{role=search,
master=true}, since its not allowed, reason:
[YES()][YES()][YES()][YES()][YES()][NO()][YES()]]; ","status":400}

Any ideas?

Thanks!

Sebastien

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hi Clinton,

Actually, what I'm saying is that I'd like every searcher to have a replica
of every shard. Right now, when I have primaries + 1 set of replicas, it
acts as I'd expect which is all the primaries on the "indexers" and all the
replicas on the "searchers". When I change the number of replicas to 3 (I
have 3 search hosts) so all my search hosts get a replica, two replicas end
up on the search hosts and 1 replica end up with the primaries on the
"indexers". Here's a sample config:

For searchers:

cluster.name: "mycluster"
node.name: ${HOSTNAME}
node.master: true
node.data: true
node.role: search
path.conf: /etc/elastic/conf
path.data: /app/elastic/data
path.work: /app/elastic/work
path.logs: /app/elastic/logs
path.plugins: /app/elastic/plugins
bootstrap.mlockall: true
transport.tcp.port: 9301
http.port: 9201
discovery.zen.minimum_master_nodes: 3
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["node0004:9301", "node0005:9301",
"node0006:9301"]
cluster.routing.allocation.awareness.attributes: role
cluster.routing.allocation.awareness.force.role.values: index,search
index.number_of_shards: 3
index.number_of_replicas: 1

For indexers:

cluster.name: "mycluster"
node.name: ${HOSTNAME}
node.master: true
node.data: true
node.role: index
path.conf: /etc/elastic/conf
path.data: /app/elastic/data
path.work: /app/elastic/work
path.logs: /app/elastic/logs
path.plugins: /app/elastic/plugins
bootstrap.mlockall: true
transport.tcp.port: 9301
http.port: 9201
discovery.zen.minimum_master_nodes: 3
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["node0004:9301", "node0005:9301",
"node0006:9301"]
cluster.routing.allocation.awareness.attributes: role
cluster.routing.allocation.awareness.force.role.values: index,search
index.number_of_shards: 3
index.number_of_replicas: 1

Thanks for the help!

Sebastien

On Sat, May 25, 2013 at 2:35 AM, Clinton Gormley clint@traveljury.comwrote:

Are you saying that the searchers have no shards? Both primaries and
replicas are on the indexers?

Please post your config (and any index settings)

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/HEzxX0bnPoo/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Hi Simon,

The configuration is pretty vanilla except the forced allocation awareness.
6 nodes from which 3 of them have the "role" of "search" and 3 of them have
the role of "index". I'm planning on pointing the "indexing clients" to the
index machines and "search clients" to the search machines.

Thanks!

Sebastien

On Sat, May 25, 2013 at 2:39 AM, simonw
simon.willnauer@elasticsearch.comwrote:

some factors in the allocation decision prevent a move of a shard here,
can you tell what kind of setting you have aside of the defaults related to
allocation and the number of machines ie. your setup?

simon

On Friday, May 24, 2013 11:01:33 PM UTC+2, Sebastien Coutu wrote:

By the way, I'm using 0.90.0 :slight_smile:

On Friday, May 24, 2013 5:01:10 PM UTC-4, Sebastien Coutu wrote:

Hi everyone,

A bit of background:

  • Cluster of six nodes, three tagged as being "indexers" and three
    tagged as being "searchers"
  • One index, three shards, replication of one.

I'd like to know if it's possible to have a replication factor of three
so all the "searchers" have a replica of all the shards. I've tried it and
both the searchers and indexers ended up with two shards each and when I
tried to move the replica shards from the "indexers" to the "searchers",
elastic returned me the following:

{"error":"**RemoteTransportException[[**node0005][inet[/ip_address:9301]][cluster/reroute]];
nested: ElasticSearchIllegalArgumentException[[move_allocation] can't
move [hopper][1], from [node0006][KylcDE8ZTL6KfETSMecxzQ][inet[
node0006/ip_address:9301]]{**role=index, master=true}, to
[node0003][PjcZhF73SxmS8G-**wQkSL0A][inet[node0003/ip_**address:9301]]{role=search,
master=true}, since its not allowed, reason: [YES()][YES()][YES()][YES()][
**YES()][NO()][YES()]]; ","status":400}

Any ideas?

Thanks!

Sebastien

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/HEzxX0bnPoo/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Sebastien, primary and replica shards are identical. When you index a
document, it hits the primary first, then the replicas after that. So all
shards are indexing shards.

Separating primaries and replicas like this is artificial and won't do what
you hope

On 27 May 2013 16:35, Sebastien Coutu scoutu@hopper.com wrote:

Hi Simon,

The configuration is pretty vanilla except the forced allocation
awareness. 6 nodes from which 3 of them have the "role" of "search" and 3
of them have the role of "index". I'm planning on pointing the "indexing
clients" to the index machines and "search clients" to the search machines.

Thanks!

Sebastien

On Sat, May 25, 2013 at 2:39 AM, simonw <simon.willnauer@elasticsearch.com

wrote:

some factors in the allocation decision prevent a move of a shard here,
can you tell what kind of setting you have aside of the defaults related to
allocation and the number of machines ie. your setup?

simon

On Friday, May 24, 2013 11:01:33 PM UTC+2, Sebastien Coutu wrote:

By the way, I'm using 0.90.0 :slight_smile:

On Friday, May 24, 2013 5:01:10 PM UTC-4, Sebastien Coutu wrote:

Hi everyone,

A bit of background:

  • Cluster of six nodes, three tagged as being "indexers" and three
    tagged as being "searchers"
  • One index, three shards, replication of one.

I'd like to know if it's possible to have a replication factor of three
so all the "searchers" have a replica of all the shards. I've tried it and
both the searchers and indexers ended up with two shards each and when I
tried to move the replica shards from the "indexers" to the "searchers",
elastic returned me the following:

{"error":"**RemoteTransportException[[**node0005][inet[/ip_address:9301]][cluster/reroute]];
nested: ElasticSearchIllegalArgumentException[[move_allocation]
can't move [hopper][1], from [node0006][**KylcDE8ZTL6KfETSMecxzQ][inet[
**node0006/ip_address:9301]]{**role=index, master=true}, to
[node0003][PjcZhF73SxmS8G-**wQkSL0A][inet[node0003/ip_**address:9301]]{role=search,
master=true}, since its not allowed, reason: [YES()][YES()][YES()][YES()][
**YES()][NO()][YES()]]; ","status":400}

Any ideas?

Thanks!

Sebastien

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/HEzxX0bnPoo/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Hi Clinton,

So I guess it's also a bad idea to have different tuning configuration for
the machines in the "search" and on the "index" groups?

Thanks,

Sebastien

On Tue, May 28, 2013 at 11:12 AM, Clinton Gormley clint@traveljury.comwrote:

Sebastien, primary and replica shards are identical. When you index a
document, it hits the primary first, then the replicas after that. So all
shards are indexing shards.

Separating primaries and replicas like this is artificial and won't do
what you hope

On 27 May 2013 16:35, Sebastien Coutu scoutu@hopper.com wrote:

Hi Simon,

The configuration is pretty vanilla except the forced allocation
awareness. 6 nodes from which 3 of them have the "role" of "search" and 3
of them have the role of "index". I'm planning on pointing the "indexing
clients" to the index machines and "search clients" to the search machines.

Thanks!

Sebastien

On Sat, May 25, 2013 at 2:39 AM, simonw <
simon.willnauer@elasticsearch.com> wrote:

some factors in the allocation decision prevent a move of a shard here,
can you tell what kind of setting you have aside of the defaults related to
allocation and the number of machines ie. your setup?

simon

On Friday, May 24, 2013 11:01:33 PM UTC+2, Sebastien Coutu wrote:

By the way, I'm using 0.90.0 :slight_smile:

On Friday, May 24, 2013 5:01:10 PM UTC-4, Sebastien Coutu wrote:

Hi everyone,

A bit of background:

  • Cluster of six nodes, three tagged as being "indexers" and three
    tagged as being "searchers"
  • One index, three shards, replication of one.

I'd like to know if it's possible to have a replication factor of
three so all the "searchers" have a replica of all the shards. I've tried
it and both the searchers and indexers ended up with two shards each and
when I tried to move the replica shards from the "indexers" to the
"searchers", elastic returned me the following:

{"error":"**RemoteTransportException[[node0005][inet[/ip_address:9301]][cluster/reroute]];
nested: ElasticSearchIllegalArgumentException[[move_allocation]
can't move [hopper][1], from [node0006][
KylcDE8ZTL6KfETSMecxzQ][inet[**node0006/ip_address:9301]]{role=index,
master=true}, to [node0003][PjcZhF73SxmS8G-
wQkSL0A][inet[node0003/ip_**address:9301]]{role=search, master=true},
since its not allowed, reason: [YES()][YES()][YES()][YES()][**YES()][NO()][YES()]];
","status":400}

Any ideas?

Thanks!

Sebastien

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/HEzxX0bnPoo/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/HEzxX0bnPoo/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.