# Troubleshooting ES Resharding. Nature of immediate tasks and other questions
**URL:** https://discuss.elastic.co/t/troubleshooting-es-resharding-nature-of-immediate-tasks-and-other-questions/23548
**Category:** Elasticsearch
**Created:** [May 4, 2015, 9:05pm UTC](https://discuss.elastic.co/t/troubleshooting-es-resharding-nature-of-immediate-tasks-and-other-questions/23548 "2015-05-04T21:05:00Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@tomas\_nino](https://discuss.elastic.co/u/tomas_nino)
#### Post date: [May 4, 2015, 9:05pm UTC](https://discuss.elastic.co/t/troubleshooting-es-resharding-nature-of-immediate-tasks-and-other-questions/23548/1 "2015-05-04T21:05:00Z")
Hi all,
We've been facing some trouble with our Elastic Search installation (v
1.5.1), mostly trying
to bring it back up. Some questions have come up.
This is our situation. We're seeing about 200 unassigned shards, which are
not being
reassigned automatically, which in turn leads our ES to move into a red
status. In this state,
queries are disabled, however we keep storing data.
What we see is that this generates refresh/update mapping tasks which are
never resolved (we think
this is because ES is in a red state). Hoping to solve this, we've been
running \_cluster/reroute on primary
shards:
curl -XPOST 'localhost:9290/\_cluster/reroute' -d '{
"commands": [
{
"allocate": {
"index": "'$INDEX'",
"shard": '$SHARD',
"node": "'$NODE'",
"allow\_primary": true
}
}
]
}'
Commands such as \_nodes/\_local/{stats, settings, os, processes} will fail
on the master node (hang indefinetly)
We monitor the pending tasks (see \_cat/pending\_tasks and \_cluster/health
output below) and see that
the IMMEDIATE tasks are queued.
However, we're wondering what's the rationale behind the queuing of this
tasks:
Is there a round robin mechanism for the IMMEDIATE or is there some way of
prioritizing the
health state of the cluster over other tasks? Will an IMMEDIATE task
preempt any other (e.g URGENT or HIGH?)
We've noticed that when queuing two IMMEDIATE tasks, the second one may
timeout if the first one
is not resolved fast enough. Is this queue being consumed by a single
thread? If so, anyway to change that?
## Thanks in advance! Tomás
\_cat/pending\_tasks
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0220771 2.1m IMMEDIATE cluster\_reroute (api)
220891 29.8s IMMEDIATE cluster\_reroute (api)
196772 10.1h HIGH update-mapping
[beer.raw.cl.business.2015-05-04][GDS\_search\_scans] / node
[v5SxZ7CdRou13tzy-N1DJg], order [1109]
220892 25.9s IMMEDIATE cluster\_reroute (api)
196773 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS\_search\_scans] / node
[BTYaSC3cT8K\_3xHDQYoNXQ], order [419]
196779 10.1h HIGH update-mapping
[beer.raw.ar.business.2015-05-04][prism\_retrieve] / node
[iDVZlJycRdeOa1PGB4Oi9Q], order [127]
220893 25.7s IMMEDIATE cluster\_reroute (api)
196787 10.1h HIGH refresh-mapping
[beer.raw.pt.business.2015-05-03][[GDS\_search\_scans]]
196786 10.1h HIGH refresh-mapping
[beer.raw.ca.business.2015-05-03][[GDS\_search\_scans]]
196774 10.1h HIGH update-mapping
[beer.raw.pe.business.2015-05-04][GDS\_search\_scans] / node
[Kx-HMg4qQKqepJb1qjjS3A], order [151]
196790 10.1h HIGH refresh-mapping
[beer.raw.ae.search.2015-05-03][[vito]]
196792 10.1h HIGH refresh-mapping
[beer.raw.tr.business.2015-05-03][[GDS\_search\_scans]]
218944 35.5m URGENT shard-started
([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K\_3xHDQYoNXQ],
relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after
recovery (replica) from node
[[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/10.70.163.240:9300]]{master=false}]]
220894 25.7s IMMEDIATE cluster\_reroute (api)
196850 10.1h HIGH refresh-mapping
[beer.raw.fi.business.2015-05-03][[GDS\_search\_scans]]
196788 10.1h HIGH refresh-mapping
[beer.raw.il.business.2015-05-03][[GDS\_search\_scans]]
196789 10.1h HIGH refresh-mapping
[beer.raw.nl.business.2015-05-03][[GDS\_search\_scans]]
Health
\_cluster/health?pretty=true"
"cluster\_name" : "beer-elastic",
"status" : "red",
"timed\_out" : false,
"number\_of\_nodes" : 20,
"number\_of\_data\_nodes" : 12,
"active\_primary\_shards" : 32192,
"active\_shards" : 64384,
"relocating\_shards" : 2,
"initializing\_shards" : 14,
"unassigned\_shards" : 182,
"number\_of\_pending\_tasks" : 13686
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)
#### Post date: [May 5, 2015, 12:42am UTC](https://discuss.elastic.co/t/troubleshooting-es-resharding-nature-of-immediate-tasks-and-other-questions/23548/2 "2015-05-05T00:42:38Z")
hello, just out of curiosity, what is the cluster index size and what is
the rationale to have so many shards?
jason
On Tue, May 5, 2015 at 5:05 AM, [tomas.nino@despegar.com](mailto:tomas.nino@despegar.com) wrote:
> Hi all,
>
> We've been facing some trouble with our Elastic Search installation (v
> 1.5.1), mostly trying
> to bring it back up. Some questions have come up.
>
> This is our situation. We're seeing about 200 unassigned shards, which are
> not being
> reassigned automatically, which in turn leads our ES to move into a red
> status. In this state,
> queries are disabled, however we keep storing data.
>
> What we see is that this generates refresh/update mapping tasks which are
> never resolved (we think
> this is because ES is in a red state). Hoping to solve this, we've been
> running \_cluster/reroute on primary
> shards:
>
> curl -XPOST 'localhost:9290/\_cluster/reroute' -d '{
> "commands": [
> {
> "allocate": {
> "index": "'$INDEX'",
> "shard": '$SHARD',
> "node": "'$NODE'",
> "allow\_primary": true
> }
> }
> ]
> }'
>
> Commands such as \_nodes/\_local/{stats, settings, os, processes} will fail
> on the master node (hang indefinetly)
>
> We monitor the pending tasks (see \_cat/pending\_tasks and \_cluster/health
> output below) and see that
> the IMMEDIATE tasks are queued.
>
> However, we're wondering what's the rationale behind the queuing of this
> tasks:
>
> Is there a round robin mechanism for the IMMEDIATE or is there some way of
> prioritizing the
> health state of the cluster over other tasks? Will an IMMEDIATE task
> preempt any other (e.g URGENT or HIGH?)
>
> We've noticed that when queuing two IMMEDIATE tasks, the second one may
> timeout if the first one
> is not resolved fast enough. Is this queue being consumed by a single
> thread? If so, anyway to change that?
>
> ## Thanks in advance! Tomás
>
> \_cat/pending\_tasks
> % Total % Received % Xferd Average Speed Time Time Time
> Current
> Dload Upload Total Spent Left
> Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
> 0220771 2.1m IMMEDIATE cluster\_reroute (api)
>
> 220891 29.8s IMMEDIATE cluster\_reroute (api)
>
> 196772 10.1h HIGH update-mapping
> [beer.raw.cl.business.2015-05-04][GDS\_search\_scans] / node
> [v5SxZ7CdRou13tzy-N1DJg], order [1109]
> 220892 25.9s IMMEDIATE cluster\_reroute (api)
>
> 196773 10.1h HIGH update-mapping
> [beer.raw.pe.business.2015-05-04][GDS\_search\_scans] / node
> [BTYaSC3cT8K\_3xHDQYoNXQ], order [419]
> 196779 10.1h HIGH update-mapping
> [beer.raw.ar.business.2015-05-04][prism\_retrieve] / node
> [iDVZlJycRdeOa1PGB4Oi9Q], order [127]
> 220893 25.7s IMMEDIATE cluster\_reroute (api)
>
> 196787 10.1h HIGH refresh-mapping
> [beer.raw.pt.business.2015-05-03][[GDS\_search\_scans]]
>
> 196786 10.1h HIGH refresh-mapping
> [beer.raw.ca.business.2015-05-03][[GDS\_search\_scans]]
>
> 196774 10.1h HIGH update-mapping
> [beer.raw.pe.business.2015-05-04][GDS\_search\_scans] / node
> [Kx-HMg4qQKqepJb1qjjS3A], order [151]
> 196790 10.1h HIGH refresh-mapping
> [beer.raw.ae.search.2015-05-03][[vito]]
>
> 196792 10.1h HIGH refresh-mapping
> [beer.raw.tr.business.2015-05-03][[GDS\_search\_scans]]
>
> 218944 35.5m URGENT shard-started
> ([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K\_3xHDQYoNXQ],
> relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after
> recovery (replica) from node
> [[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/
> 10.70.163.240:9300]]{master=false}]]
>
> 220894 25.7s IMMEDIATE cluster\_reroute (api)
>
> 196850 10.1h HIGH refresh-mapping
> [beer.raw.fi.business.2015-05-03][[GDS\_search\_scans]]
>
> 196788 10.1h HIGH refresh-mapping
> [beer.raw.il.business.2015-05-03][[GDS\_search\_scans]]
>
> 196789 10.1h HIGH refresh-mapping
> [beer.raw.nl.business.2015-05-03][[GDS\_search\_scans]]
>
> Health
>
> \_cluster/health?pretty=true"
>
> "cluster\_name" : "beer-elastic",
> "status" : "red",
> "timed\_out" : false,
> "number\_of\_nodes" : 20,
> "number\_of\_data\_nodes" : 12,
> "active\_primary\_shards" : 32192,
> "active\_shards" : 64384,
> "relocating\_shards" : 2,
> "initializing\_shards" : 14,
> "unassigned\_shards" : 182,
> "number\_of\_pending\_tasks" : 13686
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itwhNtE0bce-biK7m6MwH59yB9Q5r1tZ6CUz7U%2BjW-HTFw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itwhNtE0bce-biK7m6MwH59yB9Q5r1tZ6CUz7U%2BjW-HTFw%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 5, 2015, 1:01am UTC](https://discuss.elastic.co/t/troubleshooting-es-resharding-nature-of-immediate-tasks-and-other-questions/23548/3 "2015-05-05T01:01:14Z")
The rationale of queuing is to allow for instances where temporary load on
the cluster might otherwise reject a request.
There is no way to prioritise tasks over other tasks.
Though it looks like your problem is you are overloading your nodes. 32192
primary shards is a massive amount for only 12 nodes, you really need to
reduce this pretty dramatically to alleviate the pressure.
On 5 May 2015 at 07:05, [tomas.nino@despegar.com](mailto:tomas.nino@despegar.com) wrote:
> Hi all,
>
> We've been facing some trouble with our Elastic Search installation (v
> 1.5.1), mostly trying
> to bring it back up. Some questions have come up.
>
> This is our situation. We're seeing about 200 unassigned shards, which are
> not being
> reassigned automatically, which in turn leads our ES to move into a red
> status. In this state,
> queries are disabled, however we keep storing data.
>
> What we see is that this generates refresh/update mapping tasks which are
> never resolved (we think
> this is because ES is in a red state). Hoping to solve this, we've been
> running \_cluster/reroute on primary
> shards:
>
> curl -XPOST 'localhost:9290/\_cluster/reroute' -d '{
> "commands": [
> {
> "allocate": {
> "index": "'$INDEX'",
> "shard": '$SHARD',
> "node": "'$NODE'",
> "allow\_primary": true
> }
> }
> ]
> }'
>
> Commands such as \_nodes/\_local/{stats, settings, os, processes} will fail
> on the master node (hang indefinetly)
>
> We monitor the pending tasks (see \_cat/pending\_tasks and \_cluster/health
> output below) and see that
> the IMMEDIATE tasks are queued.
>
> However, we're wondering what's the rationale behind the queuing of this
> tasks:
>
> Is there a round robin mechanism for the IMMEDIATE or is there some way of
> prioritizing the
> health state of the cluster over other tasks? Will an IMMEDIATE task
> preempt any other (e.g URGENT or HIGH?)
>
> We've noticed that when queuing two IMMEDIATE tasks, the second one may
> timeout if the first one
> is not resolved fast enough. Is this queue being consumed by a single
> thread? If so, anyway to change that?
>
> ## Thanks in advance! Tomás
>
> \_cat/pending\_tasks
> % Total % Received % Xferd Average Speed Time Time Time
> Current
> Dload Upload Total Spent Left
> Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
> 0220771 2.1m IMMEDIATE cluster\_reroute (api)
>
> 220891 29.8s IMMEDIATE cluster\_reroute (api)
>
> 196772 10.1h HIGH update-mapping
> [beer.raw.cl.business.2015-05-04][GDS\_search\_scans] / node
> [v5SxZ7CdRou13tzy-N1DJg], order [1109]
> 220892 25.9s IMMEDIATE cluster\_reroute (api)
>
> 196773 10.1h HIGH update-mapping
> [beer.raw.pe.business.2015-05-04][GDS\_search\_scans] / node
> [BTYaSC3cT8K\_3xHDQYoNXQ], order [419]
> 196779 10.1h HIGH update-mapping
> [beer.raw.ar.business.2015-05-04][prism\_retrieve] / node
> [iDVZlJycRdeOa1PGB4Oi9Q], order [127]
> 220893 25.7s IMMEDIATE cluster\_reroute (api)
>
> 196787 10.1h HIGH refresh-mapping
> [beer.raw.pt.business.2015-05-03][[GDS\_search\_scans]]
>
> 196786 10.1h HIGH refresh-mapping
> [beer.raw.ca.business.2015-05-03][[GDS\_search\_scans]]
>
> 196774 10.1h HIGH update-mapping
> [beer.raw.pe.business.2015-05-04][GDS\_search\_scans] / node
> [Kx-HMg4qQKqepJb1qjjS3A], order [151]
> 196790 10.1h HIGH refresh-mapping
> [beer.raw.ae.search.2015-05-03][[vito]]
>
> 196792 10.1h HIGH refresh-mapping
> [beer.raw.tr.business.2015-05-03][[GDS\_search\_scans]]
>
> 218944 35.5m URGENT shard-started
> ([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K\_3xHDQYoNXQ],
> relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after
> recovery (replica) from node
> [[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/
> 10.70.163.240:9300]]{master=false}]]
>
> 220894 25.7s IMMEDIATE cluster\_reroute (api)
>
> 196850 10.1h HIGH refresh-mapping
> [beer.raw.fi.business.2015-05-03][[GDS\_search\_scans]]
>
> 196788 10.1h HIGH refresh-mapping
> [beer.raw.il.business.2015-05-03][[GDS\_search\_scans]]
>
> 196789 10.1h HIGH refresh-mapping
> [beer.raw.nl.business.2015-05-03][[GDS\_search\_scans]]
>
> Health
>
> \_cluster/health?pretty=true"
>
> "cluster\_name" : "beer-elastic",
> "status" : "red",
> "timed\_out" : false,
> "number\_of\_nodes" : 20,
> "number\_of\_data\_nodes" : 12,
> "active\_primary\_shards" : 32192,
> "active\_shards" : 64384,
> "relocating\_shards" : 2,
> "initializing\_shards" : 14,
> "unassigned\_shards" : 182,
> "number\_of\_pending\_tasks" : 13686
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8eRJr6S%2B4k65tx6sPYMeDtiEuEhxJRicGtff35Gzz%3D%2BQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8eRJr6S%2B4k65tx6sPYMeDtiEuEhxJRicGtff35Gzz%3D%2BQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Alejandro\_De\_Lio](https://discuss.elastic.co/u/Alejandro_De_Lio)
#### Post date: [May 6, 2015, 2:34pm UTC](https://discuss.elastic.co/t/troubleshooting-es-resharding-nature-of-immediate-tasks-and-other-questions/23548/4 "2015-05-06T14:34:34Z")
Hey guys, thanks for the answers.
Firstly, the index size can vary from index to index (as each index is
associated with a particular business sector, each of wich might be more or
less active). The max size is approximately 1 GB for the heaviest indexes.
Secondly, what do you think it would a sane ratio between shards and nodes
(we have 5 primary shards per index)?
Background:
- the storage nodes have 24 GB RAM and 8 cores with 500 GB attached disks
(x 12 nodes)
- we generate around 4.000 shards per day (around 400 indexes with default
configuration of 5 primary shard per index and 1 replica per shard)
- one index contains only one document type
The reason we've chosen this "many indexes design" was, fundamentally,
heterogeneous TTLs. We thought it would be better to drop a bunch of small
indexes rather than scan a whole giant index (around 3GB per shard), then
remove specific events and finally compact it.
Do you think it might be better to have just a few indexes and perform
sporadic cleanup tasks, over partitioning information into lots of
independent, small, flexible indexes? It should be taken into account that
some of the current indexes have update-mapping operations frequently
applied to it, as the document structure might have new fields.
Thanks for your time!
Ale
El lunes, 4 de mayo de 2015, 22:01:46 (UTC-3), Mark Walkom escribió:
> The rationale of queuing is to allow for instances where temporary load on
> the cluster might otherwise reject a request.
> There is no way to prioritise tasks over other tasks.
>
> Though it looks like your problem is you are overloading your nodes. 32192
> primary shards is a massive amount for only 12 nodes, you really need to
> reduce this pretty dramatically to alleviate the pressure.
>
> On 5 May 2015 at 07:05, \<[tomas...@despegar.com](mailto:tomas...@despegar.com) \\> wrote:
>
> > Hi all,
> >
> > We've been facing some trouble with our Elastic Search installation (v
> > 1.5.1), mostly trying
> > to bring it back up. Some questions have come up.
> >
> > This is our situation. We're seeing about 200 unassigned shards, which
> > are not being
> > reassigned automatically, which in turn leads our ES to move into a red
> > status. In this state,
> > queries are disabled, however we keep storing data.
> >
> > What we see is that this generates refresh/update mapping tasks which are
> > never resolved (we think
> > this is because ES is in a red state). Hoping to solve this, we've been
> > running \_cluster/reroute on primary
> > shards:
> >
> > curl -XPOST 'localhost:9290/\_cluster/reroute' -d '{
> > "commands": [
> > {
> > "allocate": {
> > "index": "'$INDEX'",
> > "shard": '$SHARD',
> > "node": "'$NODE'",
> > "allow\_primary": true
> > }
> > }
> > ]
> > }'
> >
> > Commands such as \_nodes/\_local/{stats, settings, os, processes} will fail
> > on the master node (hang indefinetly)
> >
> > We monitor the pending tasks (see \_cat/pending\_tasks and \_cluster/health
> > output below) and see that
> > the IMMEDIATE tasks are queued.
> >
> > However, we're wondering what's the rationale behind the queuing of this
> > tasks:
> >
> > Is there a round robin mechanism for the IMMEDIATE or is there some way
> > of prioritizing the
> > health state of the cluster over other tasks? Will an IMMEDIATE task
> > preempt any other (e.g URGENT or HIGH?)
> >
> > We've noticed that when queuing two IMMEDIATE tasks, the second one may
> > timeout if the first one
> > is not resolved fast enough. Is this queue being consumed by a single
> > thread? If so, anyway to change that?
> >
> > ## Thanks in advance! Tomás
> >
> > \_cat/pending\_tasks
> > % Total % Received % Xferd Average Speed Time Time Time
> > Current
> > Dload Upload Total Spent Left
> > Speed
> > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
> > 0220771 2.1m IMMEDIATE cluster\_reroute (api)
> >
> > 220891 29.8s IMMEDIATE cluster\_reroute (api)
> >
> > 196772 10.1h HIGH update-mapping
> > [beer.raw.cl.business.2015-05-04][GDS\_search\_scans] / node
> > [v5SxZ7CdRou13tzy-N1DJg], order [1109]
> > 220892 25.9s IMMEDIATE cluster\_reroute (api)
> >
> > 196773 10.1h HIGH update-mapping
> > [beer.raw.pe.business.2015-05-04][GDS\_search\_scans] / node
> > [BTYaSC3cT8K\_3xHDQYoNXQ], order [419]
> > 196779 10.1h HIGH update-mapping
> > [beer.raw.ar.business.2015-05-04][prism\_retrieve] / node
> > [iDVZlJycRdeOa1PGB4Oi9Q], order [127]
> > 220893 25.7s IMMEDIATE cluster\_reroute (api)
> >
> > 196787 10.1h HIGH refresh-mapping
> > [beer.raw.pt.business.2015-05-03][[GDS\_search\_scans]]
> >
> > 196786 10.1h HIGH refresh-mapping
> > [beer.raw.ca.business.2015-05-03][[GDS\_search\_scans]]
> >
> > 196774 10.1h HIGH update-mapping
> > [beer.raw.pe.business.2015-05-04][GDS\_search\_scans] / node
> > [Kx-HMg4qQKqepJb1qjjS3A], order [151]
> > 196790 10.1h HIGH refresh-mapping
> > [beer.raw.ae.search.2015-05-03][[vito]]
> >
> > 196792 10.1h HIGH refresh-mapping
> > [beer.raw.tr.business.2015-05-03][[GDS\_search\_scans]]
> >
> > 218944 35.5m URGENT shard-started
> > ([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K\_3xHDQYoNXQ],
> > relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after
> > recovery (replica) from node
> > [[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/
> > 10.70.163.240:9300]]{master=false}]]
> >
> > 220894 25.7s IMMEDIATE cluster\_reroute (api)
> >
> > 196850 10.1h HIGH refresh-mapping
> > [beer.raw.fi.business.2015-05-03][[GDS\_search\_scans]]
> >
> > 196788 10.1h HIGH refresh-mapping
> > [beer.raw.il.business.2015-05-03][[GDS\_search\_scans]]
> >
> > 196789 10.1h HIGH refresh-mapping
> > [beer.raw.nl.business.2015-05-03][[GDS\_search\_scans]]
> >
> > Health
> >
> > \_cluster/health?pretty=true"
> >
> > "cluster\_name" : "beer-elastic",
> > "status" : "red",
> > "timed\_out" : false,
> > "number\_of\_nodes" : 20,
> > "number\_of\_data\_nodes" : 12,
> > "active\_primary\_shards" : 32192,
> > "active\_shards" : 64384,
> > "relocating\_shards" : 2,
> > "initializing\_shards" : 14,
> > "unassigned\_shards" : 182,
> > "number\_of\_pending\_tasks" : 13686
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
## -- Please update your bookmarks! We moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/59371740-cc04-436d-8e78-4daa6822a83b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/59371740-cc04-436d-8e78-4daa6822a83b%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 11, 2015, 11:37pm UTC](https://discuss.elastic.co/t/troubleshooting-es-resharding-nature-of-immediate-tasks-and-other-questions/23548/5 "2015-05-11T23:37:13Z")
If your largest index is only 1GB, then try reducing that to a single shard
(with one replica) and see how the performance goes.
I'd trial that on a few indices before rolling it out across the board
though.
Again, the core issue is that you have too many shards!
On 7 May 2015 at 00:34, Alejandro De Lío [alejandro.delio@gmail.com](mailto:alejandro.delio@gmail.com) wrote:
> Hey guys, thanks for the answers.
>
> Firstly, the index size can vary from index to index (as each index is
> associated with a particular business sector, each of wich might be more or
> less active). The max size is approximately 1 GB for the heaviest indexes.
>
> Secondly, what do you think it would a sane ratio between shards and nodes
> (we have 5 primary shards per index)?
>
> Background:
>
> - the storage nodes have 24 GB RAM and 8 cores with 500 GB attached disks
> (x 12 nodes)
> - we generate around 4.000 shards per day (around 400 indexes with default
> configuration of 5 primary shard per index and 1 replica per shard)
> - one index contains only one document type
>
> The reason we've chosen this "many indexes design" was, fundamentally,
> heterogeneous TTLs. We thought it would be better to drop a bunch of small
> indexes rather than scan a whole giant index (around 3GB per shard), then
> remove specific events and finally compact it.
>
> Do you think it might be better to have just a few indexes and perform
> sporadic cleanup tasks, over partitioning information into lots of
> independent, small, flexible indexes? It should be taken into account that
> some of the current indexes have update-mapping operations frequently
> applied to it, as the document structure might have new fields.
>
> Thanks for your time!
> Ale
>
> El lunes, 4 de mayo de 2015, 22:01:46 (UTC-3), Mark Walkom escribió:
>
> > The rationale of queuing is to allow for instances where temporary load
> > on the cluster might otherwise reject a request.
> > There is no way to prioritise tasks over other tasks.
> >
> > Though it looks like your problem is you are overloading your nodes.
> > 32192 primary shards is a massive amount for only 12 nodes, you really need
> > to reduce this pretty dramatically to alleviate the pressure.
> >
> > On 5 May 2015 at 07:05, [tomas...@despegar.com](mailto:tomas...@despegar.com) wrote:
> >
> > > Hi all,
> > >
> > > We've been facing some trouble with our Elastic Search installation (v
> > > 1.5.1), mostly trying
> > > to bring it back up. Some questions have come up.
> > >
> > > This is our situation. We're seeing about 200 unassigned shards, which
> > > are not being
> > > reassigned automatically, which in turn leads our ES to move into a red
> > > status. In this state,
> > > queries are disabled, however we keep storing data.
> > >
> > > What we see is that this generates refresh/update mapping tasks which
> > > are never resolved (we think
> > > this is because ES is in a red state). Hoping to solve this, we've been
> > > running \_cluster/reroute on primary
> > > shards:
> > >
> > > curl -XPOST 'localhost:9290/\_cluster/reroute' -d '{
> > > "commands": [
> > > {
> > > "allocate": {
> > > "index": "'$INDEX'",
> > > "shard": '$SHARD',
> > > "node": "'$NODE'",
> > > "allow\_primary": true
> > > }
> > > }
> > > ]
> > > }'
> > >
> > > Commands such as \_nodes/\_local/{stats, settings, os, processes} will
> > > fail on the master node (hang indefinetly)
> > >
> > > We monitor the pending tasks (see \_cat/pending\_tasks and
> > > \_cluster/health output below) and see that
> > > the IMMEDIATE tasks are queued.
> > >
> > > However, we're wondering what's the rationale behind the queuing of this
> > > tasks:
> > >
> > > Is there a round robin mechanism for the IMMEDIATE or is there some way
> > > of prioritizing the
> > > health state of the cluster over other tasks? Will an IMMEDIATE task
> > > preempt any other (e.g URGENT or HIGH?)
> > >
> > > We've noticed that when queuing two IMMEDIATE tasks, the second one may
> > > timeout if the first one
> > > is not resolved fast enough. Is this queue being consumed by a single
> > > thread? If so, anyway to change that?
> > >
> > > ## Thanks in advance! Tomás
> > >
> > > \_cat/pending\_tasks
> > > % Total % Received % Xferd Average Speed Time Time Time
> > > Current
> > > Dload Upload Total Spent Left
> > > Speed
> > > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
> > > 0220771 2.1m IMMEDIATE cluster\_reroute (api)
> > >
> > > 220891 29.8s IMMEDIATE cluster\_reroute (api)
> > >
> > > 196772 10.1h HIGH update-mapping
> > > [beer.raw.cl.business.2015-05-04][GDS\_search\_scans] / node
> > > [v5SxZ7CdRou13tzy-N1DJg], order [1109]
> > > 220892 25.9s IMMEDIATE cluster\_reroute (api)
> > >
> > > 196773 10.1h HIGH update-mapping
> > > [beer.raw.pe.business.2015-05-04][GDS\_search\_scans] / node
> > > [BTYaSC3cT8K\_3xHDQYoNXQ], order [419]
> > > 196779 10.1h HIGH update-mapping
> > > [beer.raw.ar.business.2015-05-04][prism\_retrieve] / node
> > > [iDVZlJycRdeOa1PGB4Oi9Q], order [127]
> > > 220893 25.7s IMMEDIATE cluster\_reroute (api)
> > >
> > > 196787 10.1h HIGH refresh-mapping
> > > [beer.raw.pt.business.2015-05-03][[GDS\_search\_scans]]
> > >
> > > 196786 10.1h HIGH refresh-mapping
> > > [beer.raw.ca.business.2015-05-03][[GDS\_search\_scans]]
> > >
> > > 196774 10.1h HIGH update-mapping
> > > [beer.raw.pe.business.2015-05-04][GDS\_search\_scans] / node
> > > [Kx-HMg4qQKqepJb1qjjS3A], order [151]
> > > 196790 10.1h HIGH refresh-mapping
> > > [beer.raw.ae.search.2015-05-03][[vito]]
> > >
> > > 196792 10.1h HIGH refresh-mapping
> > > [beer.raw.tr.business.2015-05-03][[GDS\_search\_scans]]
> > >
> > > 218944 35.5m URGENT shard-started
> > > ([beer.raw.gy.performance.2015-04-07][2], node[BTYaSC3cT8K\_3xHDQYoNXQ],
> > > relocating [0clH8MU6Q5Wt8phPRbuTLg], [P], s[INITIALIZING]), reason [after
> > > recovery (replica) from node
> > > [[beer-elastic-s-08][0clH8MU6Q5Wt8phPRbuTLg][beer-elastic-s-08][inet[/
> > > 10.70.163.240:9300]]{master=false}]]
> > >
> > > 220894 25.7s IMMEDIATE cluster\_reroute (api)
> > >
> > > 196850 10.1h HIGH refresh-mapping
> > > [beer.raw.fi.business.2015-05-03][[GDS\_search\_scans]]
> > >
> > > 196788 10.1h HIGH refresh-mapping
> > > [beer.raw.il.business.2015-05-03][[GDS\_search\_scans]]
> > >
> > > 196789 10.1h HIGH refresh-mapping
> > > [beer.raw.nl.business.2015-05-03][[GDS\_search\_scans]]
> > >
> > > Health
> > >
> > > \_cluster/health?pretty=true"
> > >
> > > "cluster\_name" : "beer-elastic",
> > > "status" : "red",
> > > "timed\_out" : false,
> > > "number\_of\_nodes" : 20,
> > > "number\_of\_data\_nodes" : 12,
> > > "active\_primary\_shards" : 32192,
> > > "active\_shards" : 64384,
> > > "relocating\_shards" : 2,
> > > "initializing\_shards" : 14,
> > > "unassigned\_shards" : 182,
> > > "number\_of\_pending\_tasks" : 13686
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e3f4f449-d359-4ef1-a6a4-a445bb916371%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > Please update your bookmarks! We moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
>
> * * *
>
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/59371740-cc04-436d-8e78-4daa6822a83b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/59371740-cc04-436d-8e78-4daa6822a83b%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/59371740-cc04-436d-8e78-4daa6822a83b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/59371740-cc04-436d-8e78-4daa6822a83b%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8dTdx6n8RfqMwgUX\_\_%2B1\_3FnMru%3DCGF4SPREy-FbGiFg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8dTdx6n8RfqMwgUX__%2B1_3FnMru%3DCGF4SPREy-FbGiFg%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:14am UTC](https://discuss.elastic.co/t/troubleshooting-es-resharding-nature-of-immediate-tasks-and-other-questions/23548/6 "2017-07-06T00:14:32Z")