Troubleshooting ES Resharding. Nature of immediate tasks and other questions

Elastic Stack Elasticsearch
1

Hi all,

We've been facing some trouble with our Elastic Search installation (v
1.5.1), mostly trying
to bring it back up. Some questions have come up.

This is our situation. We're seeing about 200 unassigned shards, which are
not being
reassigned automatically, which in turn leads our ES to move into a red
status. In this state,
queries are disabled, however we keep storing data.

What we see is that this generates refresh/update mapping tasks which are
never resolved (we think
this is because ES is in a red state). Hoping to solve this, we've been
running _cluster/reroute on primary
shards:

curl -XPOST 'localhost:9290/_cluster/reroute' -d '{
"commands": [
{
"allocate": {
"index": "'$INDEX'",
"shard": '$SHARD',
"node": "'$NODE'",
"allow_primary": true
}
}
]
}'

Commands such as _nodes/_local/{stats, settings, os, processes} will fail
on the master node (hang indefinetly)

We monitor the pending tasks (see _cat/pending_tasks and _cluster/health
output below) and see that
the IMMEDIATE tasks are queued.

However, we're wondering what's the rationale behind the queuing of this
tasks:

Is there a round robin mechanism for the IMMEDIATE or is there some way of
prioritizing the
health state of the cluster over other tasks? Will an IMMEDIATE task
preempt any other (e.g URGENT or HIGH?)

We've noticed that when queuing two IMMEDIATE tasks, the second one may
timeout if the first one
is not resolved fast enough. Is this queue being consumed by a single
thread? If so, anyway to change that?

Thanks in advance!
Tomás

_cat/pending_tasks
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0220771 2.1m IMMEDIATE cluster_reroute (api)

220891 29.8s IMMEDIATE cluster_reroute (api)

196772 10.1h HIGH update-mapping
[beer.raw.cl.business.2015-05-04][GDS_search_scans] / node
[v5SxZ7CdRou13tzy-N1DJg], order [1109]
220892 25.9s IMMEDIATE cluster_reroute (api)

196773 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS_search_scans] / node
[BTYaSC3cT8K_3xHDQYoNXQ], order [419]
196779 10.1h HIGH update-mapping
[beer.raw.ar.business.2015-05-04][prism_retrieve] / node
[iDVZlJycRdeOa1PGB4Oi9Q], order [127]
220893 25.7s IMMEDIATE cluster_reroute (api)

196787 10.1h HIGH refresh-mapping
[beer.raw.pt.business.2015-05-03][[GDS_search_scans]]

196786 10.1h HIGH refresh-mapping
[beer.raw.ca.business.2015-05-03][[GDS_search_scans]]

196774 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS_search_scans] / node
[Kx-HMg4qQKqepJb1qjjS3A], order [151]
196790 10.1h HIGH refresh-mapping
[beer.raw.ae.search.2015-05-03][[vito]]

196792 10.1h HIGH refresh-mapping
[beer.raw.tr.business.2015-05-03][[GDS_search_scans]]

218944 35.5m URGENT shard-started
([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K_3xHDQYoNXQ],
relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after
recovery (replica) from node
[[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/10.70.163.240:9300]]{master=false}]]

220894 25.7s IMMEDIATE cluster_reroute (api)

196850 10.1h HIGH refresh-mapping
[beer.raw.fi.business.2015-05-03][[GDS_search_scans]]

196788 10.1h HIGH refresh-mapping
[beer.raw.il.business.2015-05-03][[GDS_search_scans]]

196789 10.1h HIGH refresh-mapping
[beer.raw.nl.business.2015-05-03][[GDS_search_scans]]

Health

_cluster/health?pretty=true"

"cluster_name" : "beer-elastic",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 20,
"number_of_data_nodes" : 12,
"active_primary_shards" : 32192,
"active_shards" : 64384,
"relocating_shards" : 2,
"initializing_shards" : 14,
"unassigned_shards" : 182,
"number_of_pending_tasks" : 13686

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

hello, just out of curiosity, what is the cluster index size and what is
the rationale to have so many shards?

jason

On Tue, May 5, 2015 at 5:05 AM, tomas.nino@despegar.com wrote:

Hi all,

We've been facing some trouble with our Elastic Search installation (v
1.5.1), mostly trying
to bring it back up. Some questions have come up.

This is our situation. We're seeing about 200 unassigned shards, which are
not being
reassigned automatically, which in turn leads our ES to move into a red
status. In this state,
queries are disabled, however we keep storing data.

What we see is that this generates refresh/update mapping tasks which are
never resolved (we think
this is because ES is in a red state). Hoping to solve this, we've been
running _cluster/reroute on primary
shards:

curl -XPOST 'localhost:9290/_cluster/reroute' -d '{
"commands": [
{
"allocate": {
"index": "'$INDEX'",
"shard": '$SHARD',
"node": "'$NODE'",
"allow_primary": true
}
}
]
}'

Commands such as _nodes/_local/{stats, settings, os, processes} will fail
on the master node (hang indefinetly)

We monitor the pending tasks (see _cat/pending_tasks and _cluster/health
output below) and see that
the IMMEDIATE tasks are queued.

However, we're wondering what's the rationale behind the queuing of this
tasks:

Is there a round robin mechanism for the IMMEDIATE or is there some way of
prioritizing the
health state of the cluster over other tasks? Will an IMMEDIATE task
preempt any other (e.g URGENT or HIGH?)

We've noticed that when queuing two IMMEDIATE tasks, the second one may
timeout if the first one
is not resolved fast enough. Is this queue being consumed by a single
thread? If so, anyway to change that?

Thanks in advance!
Tomás

_cat/pending_tasks
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0220771 2.1m IMMEDIATE cluster_reroute (api)

220891 29.8s IMMEDIATE cluster_reroute (api)

196772 10.1h HIGH update-mapping
[beer.raw.cl.business.2015-05-04][GDS_search_scans] / node
[v5SxZ7CdRou13tzy-N1DJg], order [1109]
220892 25.9s IMMEDIATE cluster_reroute (api)

196773 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS_search_scans] / node
[BTYaSC3cT8K_3xHDQYoNXQ], order [419]
196779 10.1h HIGH update-mapping
[beer.raw.ar.business.2015-05-04][prism_retrieve] / node
[iDVZlJycRdeOa1PGB4Oi9Q], order [127]
220893 25.7s IMMEDIATE cluster_reroute (api)

196787 10.1h HIGH refresh-mapping
[beer.raw.pt.business.2015-05-03][[GDS_search_scans]]

196786 10.1h HIGH refresh-mapping
[beer.raw.ca.business.2015-05-03][[GDS_search_scans]]

196774 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS_search_scans] / node
[Kx-HMg4qQKqepJb1qjjS3A], order [151]
196790 10.1h HIGH refresh-mapping
[beer.raw.ae.search.2015-05-03][[vito]]

196792 10.1h HIGH refresh-mapping
[beer.raw.tr.business.2015-05-03][[GDS_search_scans]]

218944 35.5m URGENT shard-started
([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K_3xHDQYoNXQ],
relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after
recovery (replica) from node
[[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/
10.70.163.240:9300]]{master=false}]]

220894 25.7s IMMEDIATE cluster_reroute (api)

196850 10.1h HIGH refresh-mapping
[beer.raw.fi.business.2015-05-03][[GDS_search_scans]]

196788 10.1h HIGH refresh-mapping
[beer.raw.il.business.2015-05-03][[GDS_search_scans]]

196789 10.1h HIGH refresh-mapping
[beer.raw.nl.business.2015-05-03][[GDS_search_scans]]

Health

_cluster/health?pretty=true"

"cluster_name" : "beer-elastic",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 20,
"number_of_data_nodes" : 12,
"active_primary_shards" : 32192,
"active_shards" : 64384,
"relocating_shards" : 2,
"initializing_shards" : 14,
"unassigned_shards" : 182,
"number_of_pending_tasks" : 13686

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itwhNtE0bce-biK7m6MwH59yB9Q5r1tZ6CUz7U%2BjW-HTFw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

The rationale of queuing is to allow for instances where temporary load on
the cluster might otherwise reject a request.
There is no way to prioritise tasks over other tasks.

Though it looks like your problem is you are overloading your nodes. 32192
primary shards is a massive amount for only 12 nodes, you really need to
reduce this pretty dramatically to alleviate the pressure.

On 5 May 2015 at 07:05, tomas.nino@despegar.com wrote:

Hi all,

We've been facing some trouble with our Elastic Search installation (v
1.5.1), mostly trying
to bring it back up. Some questions have come up.

This is our situation. We're seeing about 200 unassigned shards, which are
not being
reassigned automatically, which in turn leads our ES to move into a red
status. In this state,
queries are disabled, however we keep storing data.

What we see is that this generates refresh/update mapping tasks which are
never resolved (we think
this is because ES is in a red state). Hoping to solve this, we've been
running _cluster/reroute on primary
shards:

curl -XPOST 'localhost:9290/_cluster/reroute' -d '{
"commands": [
{
"allocate": {
"index": "'$INDEX'",
"shard": '$SHARD',
"node": "'$NODE'",
"allow_primary": true
}
}
]
}'

Commands such as _nodes/_local/{stats, settings, os, processes} will fail
on the master node (hang indefinetly)

We monitor the pending tasks (see _cat/pending_tasks and _cluster/health
output below) and see that
the IMMEDIATE tasks are queued.

However, we're wondering what's the rationale behind the queuing of this
tasks:

Is there a round robin mechanism for the IMMEDIATE or is there some way of
prioritizing the
health state of the cluster over other tasks? Will an IMMEDIATE task
preempt any other (e.g URGENT or HIGH?)

We've noticed that when queuing two IMMEDIATE tasks, the second one may
timeout if the first one
is not resolved fast enough. Is this queue being consumed by a single
thread? If so, anyway to change that?

Thanks in advance!
Tomás

_cat/pending_tasks
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0220771 2.1m IMMEDIATE cluster_reroute (api)

220891 29.8s IMMEDIATE cluster_reroute (api)

196772 10.1h HIGH update-mapping
[beer.raw.cl.business.2015-05-04][GDS_search_scans] / node
[v5SxZ7CdRou13tzy-N1DJg], order [1109]
220892 25.9s IMMEDIATE cluster_reroute (api)

196773 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS_search_scans] / node
[BTYaSC3cT8K_3xHDQYoNXQ], order [419]
196779 10.1h HIGH update-mapping
[beer.raw.ar.business.2015-05-04][prism_retrieve] / node
[iDVZlJycRdeOa1PGB4Oi9Q], order [127]
220893 25.7s IMMEDIATE cluster_reroute (api)

196787 10.1h HIGH refresh-mapping
[beer.raw.pt.business.2015-05-03][[GDS_search_scans]]

196786 10.1h HIGH refresh-mapping
[beer.raw.ca.business.2015-05-03][[GDS_search_scans]]

196774 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS_search_scans] / node
[Kx-HMg4qQKqepJb1qjjS3A], order [151]
196790 10.1h HIGH refresh-mapping
[beer.raw.ae.search.2015-05-03][[vito]]

196792 10.1h HIGH refresh-mapping
[beer.raw.tr.business.2015-05-03][[GDS_search_scans]]

218944 35.5m URGENT shard-started
([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K_3xHDQYoNXQ],
relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after
recovery (replica) from node
[[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/
10.70.163.240:9300]]{master=false}]]

220894 25.7s IMMEDIATE cluster_reroute (api)

196850 10.1h HIGH refresh-mapping
[beer.raw.fi.business.2015-05-03][[GDS_search_scans]]

196788 10.1h HIGH refresh-mapping
[beer.raw.il.business.2015-05-03][[GDS_search_scans]]

196789 10.1h HIGH refresh-mapping
[beer.raw.nl.business.2015-05-03][[GDS_search_scans]]

Health

_cluster/health?pretty=true"

"cluster_name" : "beer-elastic",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 20,
"number_of_data_nodes" : 12,
"active_primary_shards" : 32192,
"active_shards" : 64384,
"relocating_shards" : 2,
"initializing_shards" : 14,
"unassigned_shards" : 182,
"number_of_pending_tasks" : 13686

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8eRJr6S%2B4k65tx6sPYMeDtiEuEhxJRicGtff35Gzz%3D%2BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

Hey guys, thanks for the answers.

Firstly, the index size can vary from index to index (as each index is
associated with a particular business sector, each of wich might be more or
less active). The max size is approximately 1 GB for the heaviest indexes.

Secondly, what do you think it would a sane ratio between shards and nodes
(we have 5 primary shards per index)?

Background:

  • the storage nodes have 24 GB RAM and 8 cores with 500 GB attached disks
    (x 12 nodes)
  • we generate around 4.000 shards per day (around 400 indexes with default
    configuration of 5 primary shard per index and 1 replica per shard)
  • one index contains only one document type

The reason we've chosen this "many indexes design" was, fundamentally,
heterogeneous TTLs. We thought it would be better to drop a bunch of small
indexes rather than scan a whole giant index (around 3GB per shard), then
remove specific events and finally compact it.

Do you think it might be better to have just a few indexes and perform
sporadic cleanup tasks, over partitioning information into lots of
independent, small, flexible indexes? It should be taken into account that
some of the current indexes have update-mapping operations frequently
applied to it, as the document structure might have new fields.

Thanks for your time!
Ale

El lunes, 4 de mayo de 2015, 22:01:46 (UTC-3), Mark Walkom escribió:

The rationale of queuing is to allow for instances where temporary load on
the cluster might otherwise reject a request.
There is no way to prioritise tasks over other tasks.

Though it looks like your problem is you are overloading your nodes. 32192
primary shards is a massive amount for only 12 nodes, you really need to
reduce this pretty dramatically to alleviate the pressure.

On 5 May 2015 at 07:05, <tomas...@despegar.com <javascript:>> wrote:

Hi all,

We've been facing some trouble with our Elastic Search installation (v
1.5.1), mostly trying
to bring it back up. Some questions have come up.

This is our situation. We're seeing about 200 unassigned shards, which
are not being
reassigned automatically, which in turn leads our ES to move into a red
status. In this state,
queries are disabled, however we keep storing data.

What we see is that this generates refresh/update mapping tasks which are
never resolved (we think
this is because ES is in a red state). Hoping to solve this, we've been
running _cluster/reroute on primary
shards:

curl -XPOST 'localhost:9290/_cluster/reroute' -d '{
"commands": [
{
"allocate": {
"index": "'$INDEX'",
"shard": '$SHARD',
"node": "'$NODE'",
"allow_primary": true
}
}
]
}'

Commands such as _nodes/_local/{stats, settings, os, processes} will fail
on the master node (hang indefinetly)

We monitor the pending tasks (see _cat/pending_tasks and _cluster/health
output below) and see that
the IMMEDIATE tasks are queued.

However, we're wondering what's the rationale behind the queuing of this
tasks:

Is there a round robin mechanism for the IMMEDIATE or is there some way
of prioritizing the
health state of the cluster over other tasks? Will an IMMEDIATE task
preempt any other (e.g URGENT or HIGH?)

We've noticed that when queuing two IMMEDIATE tasks, the second one may
timeout if the first one
is not resolved fast enough. Is this queue being consumed by a single
thread? If so, anyway to change that?

Thanks in advance!
Tomás

_cat/pending_tasks
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0220771 2.1m IMMEDIATE cluster_reroute (api)

220891 29.8s IMMEDIATE cluster_reroute (api)

196772 10.1h HIGH update-mapping
[beer.raw.cl.business.2015-05-04][GDS_search_scans] / node
[v5SxZ7CdRou13tzy-N1DJg], order [1109]
220892 25.9s IMMEDIATE cluster_reroute (api)

196773 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS_search_scans] / node
[BTYaSC3cT8K_3xHDQYoNXQ], order [419]
196779 10.1h HIGH update-mapping
[beer.raw.ar.business.2015-05-04][prism_retrieve] / node
[iDVZlJycRdeOa1PGB4Oi9Q], order [127]
220893 25.7s IMMEDIATE cluster_reroute (api)

196787 10.1h HIGH refresh-mapping
[beer.raw.pt.business.2015-05-03][[GDS_search_scans]]

196786 10.1h HIGH refresh-mapping
[beer.raw.ca.business.2015-05-03][[GDS_search_scans]]

196774 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS_search_scans] / node
[Kx-HMg4qQKqepJb1qjjS3A], order [151]
196790 10.1h HIGH refresh-mapping
[beer.raw.ae.search.2015-05-03][[vito]]

196792 10.1h HIGH refresh-mapping
[beer.raw.tr.business.2015-05-03][[GDS_search_scans]]

218944 35.5m URGENT shard-started
([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K_3xHDQYoNXQ],
relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after
recovery (replica) from node
[[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/
10.70.163.240:9300]]{master=false}]]

220894 25.7s IMMEDIATE cluster_reroute (api)

196850 10.1h HIGH refresh-mapping
[beer.raw.fi.business.2015-05-03][[GDS_search_scans]]

196788 10.1h HIGH refresh-mapping
[beer.raw.il.business.2015-05-03][[GDS_search_scans]]

196789 10.1h HIGH refresh-mapping
[beer.raw.nl.business.2015-05-03][[GDS_search_scans]]

Health

_cluster/health?pretty=true"

"cluster_name" : "beer-elastic",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 20,
"number_of_data_nodes" : 12,
"active_primary_shards" : 32192,
"active_shards" : 64384,
"relocating_shards" : 2,
"initializing_shards" : 14,
"unassigned_shards" : 182,
"number_of_pending_tasks" : 13686

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/59371740-cc04-436d-8e78-4daa6822a83b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

If your largest index is only 1GB, then try reducing that to a single shard
(with one replica) and see how the performance goes.
I'd trial that on a few indices before rolling it out across the board
though.

Again, the core issue is that you have too many shards!

On 7 May 2015 at 00:34, Alejandro De Lío alejandro.delio@gmail.com wrote:

Hey guys, thanks for the answers.

Firstly, the index size can vary from index to index (as each index is
associated with a particular business sector, each of wich might be more or
less active). The max size is approximately 1 GB for the heaviest indexes.

Secondly, what do you think it would a sane ratio between shards and nodes
(we have 5 primary shards per index)?

Background:

  • the storage nodes have 24 GB RAM and 8 cores with 500 GB attached disks
    (x 12 nodes)
  • we generate around 4.000 shards per day (around 400 indexes with default
    configuration of 5 primary shard per index and 1 replica per shard)
  • one index contains only one document type

The reason we've chosen this "many indexes design" was, fundamentally,
heterogeneous TTLs. We thought it would be better to drop a bunch of small
indexes rather than scan a whole giant index (around 3GB per shard), then
remove specific events and finally compact it.

Do you think it might be better to have just a few indexes and perform
sporadic cleanup tasks, over partitioning information into lots of
independent, small, flexible indexes? It should be taken into account that
some of the current indexes have update-mapping operations frequently
applied to it, as the document structure might have new fields.

Thanks for your time!
Ale

El lunes, 4 de mayo de 2015, 22:01:46 (UTC-3), Mark Walkom escribió:

The rationale of queuing is to allow for instances where temporary load
on the cluster might otherwise reject a request.
There is no way to prioritise tasks over other tasks.

Though it looks like your problem is you are overloading your nodes.
32192 primary shards is a massive amount for only 12 nodes, you really need
to reduce this pretty dramatically to alleviate the pressure.

On 5 May 2015 at 07:05, tomas...@despegar.com wrote:

Hi all,

We've been facing some trouble with our Elastic Search installation (v
1.5.1), mostly trying
to bring it back up. Some questions have come up.

This is our situation. We're seeing about 200 unassigned shards, which
are not being
reassigned automatically, which in turn leads our ES to move into a red
status. In this state,
queries are disabled, however we keep storing data.

What we see is that this generates refresh/update mapping tasks which
are never resolved (we think
this is because ES is in a red state). Hoping to solve this, we've been
running _cluster/reroute on primary
shards:

curl -XPOST 'localhost:9290/_cluster/reroute' -d '{
"commands": [
{
"allocate": {
"index": "'$INDEX'",
"shard": '$SHARD',
"node": "'$NODE'",
"allow_primary": true
}
}
]
}'

Commands such as _nodes/_local/{stats, settings, os, processes} will
fail on the master node (hang indefinetly)

We monitor the pending tasks (see _cat/pending_tasks and
_cluster/health output below) and see that
the IMMEDIATE tasks are queued.

However, we're wondering what's the rationale behind the queuing of this
tasks:

Is there a round robin mechanism for the IMMEDIATE or is there some way
of prioritizing the
health state of the cluster over other tasks? Will an IMMEDIATE task
preempt any other (e.g URGENT or HIGH?)

We've noticed that when queuing two IMMEDIATE tasks, the second one may
timeout if the first one
is not resolved fast enough. Is this queue being consumed by a single
thread? If so, anyway to change that?

Thanks in advance!
Tomás

_cat/pending_tasks
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0220771 2.1m IMMEDIATE cluster_reroute (api)

220891 29.8s IMMEDIATE cluster_reroute (api)

196772 10.1h HIGH update-mapping
[beer.raw.cl.business.2015-05-04][GDS_search_scans] / node
[v5SxZ7CdRou13tzy-N1DJg], order [1109]
220892 25.9s IMMEDIATE cluster_reroute (api)

196773 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS_search_scans] / node
[BTYaSC3cT8K_3xHDQYoNXQ], order [419]
196779 10.1h HIGH update-mapping
[beer.raw.ar.business.2015-05-04][prism_retrieve] / node
[iDVZlJycRdeOa1PGB4Oi9Q], order [127]
220893 25.7s IMMEDIATE cluster_reroute (api)

196787 10.1h HIGH refresh-mapping
[beer.raw.pt.business.2015-05-03][[GDS_search_scans]]

196786 10.1h HIGH refresh-mapping
[beer.raw.ca.business.2015-05-03][[GDS_search_scans]]

196774 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS_search_scans] / node
[Kx-HMg4qQKqepJb1qjjS3A], order [151]
196790 10.1h HIGH refresh-mapping
[beer.raw.ae.search.2015-05-03][[vito]]

196792 10.1h HIGH refresh-mapping
[beer.raw.tr.business.2015-05-03][[GDS_search_scans]]

218944 35.5m URGENT shard-started
([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K_3xHDQYoNXQ],
relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after
recovery (replica) from node
[[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/
10.70.163.240:9300]]{master=false}]]

220894 25.7s IMMEDIATE cluster_reroute (api)

196850 10.1h HIGH refresh-mapping
[beer.raw.fi.business.2015-05-03][[GDS_search_scans]]

196788 10.1h HIGH refresh-mapping
[beer.raw.il.business.2015-05-03][[GDS_search_scans]]

196789 10.1h HIGH refresh-mapping
[beer.raw.nl.business.2015-05-03][[GDS_search_scans]]

Health

_cluster/health?pretty=true"

"cluster_name" : "beer-elastic",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 20,
"number_of_data_nodes" : 12,
"active_primary_shards" : 32192,
"active_shards" : 64384,
"relocating_shards" : 2,
"initializing_shards" : 14,
"unassigned_shards" : 182,
"number_of_pending_tasks" : 13686

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/59371740-cc04-436d-8e78-4daa6822a83b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/59371740-cc04-436d-8e78-4daa6822a83b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8dTdx6n8RfqMwgUX__%2B1_3FnMru%3DCGF4SPREy-FbGiFg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.