# Replica stuck in relocation, high cpu usage **URL:** \ **Category:** Elasticsearch\ **Created:** [May 15, 2013, 9:28am UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970 "2013-05-15T09:28:43Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Bill\_de\_hOra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bill_de_hora/32/2059_2.png) [@Bill\_de\_hOra](https://discuss.elastic.co/u/Bill_de_hOra)\ **Post date:** [May 15, 2013, 9:28am UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/1 "2013-05-15T09:28:43Z")
We experienced a master re-election, seemingly due to a timeout (or underlying throttling of the ec2 discovery mechanism in AWS, we're working through that with Amazon) . Regardless, the previous master got into a bad state similar to [https://github.com/elasticsearch/elasticsearch/issues/3017](https://github.com/elasticsearch/elasticsearch/issues/3017) (excised from cluster, still reporting green health, needed a restart to join). As a result or related to this, I'd has one replica in a relocating state for about 18 hours. I let it run to see if it would complete, but I'm wondering if it will at this point. { "cluster\_name" : "es\_cayovaprod\_0001", "status" : "yellow", "timed\_out" : false, "number\_of\_nodes" : 19, "number\_of\_data\_nodes" : 6, "active\_primary\_shards" : 101, "active\_shards" : 292, "relocating\_shards" : 1, "initializing\_shards" : 1, "unassigned\_shards" : 0 } According to big desk, the cluster state for the replica/shard is - ip-10-239-110-94: replica, relocating ip-10-65-49-148: replica, recovering ip-10-36-174-153: primary, green ip-10-36-201-76: replica, green The index in question is configured to have 2 replicas. The first two nodes in transition show a high cpu relative to the other nodes ([https://pbs.twimg.com/media/BKS1BC-CIAE5h\_\_.png:large](https://pbs.twimg.com/media/BKS1BC-CIAE5h__.png:large)), according to htop they're sitting at about 33% which doesn't seem unreasonable in itself (the length of time at that level is the concern) . The log for ip-10-239-110-94 around the time of the start of the CPU burst - """ [2013-05-14 15:18:18,219][WARN][indices.cluster] [ip-10-239-110-94] [objects\_0001][22] master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}] marked shard as started, but shard have not been created, mark shard as failed [2013-05-14 15:18:18,219][WARN][cluster.action.shard] [ip-10-239-110-94] sending failed shard for [objects\_0001][22], node[ybnrwqQ8RPCa69L4SqE7bA], [P], s[STARTED], reason [master [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a} marked shard as started, but shard have not been created, mark shard as failed] [2013-05-14 15:54:03,981][DEBUG][discovery.zen.fd] [ip-10-239-110-94] [master] restarting fault detection against master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}], reason [new cluster stare received and we monitor the wrong master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]] [2013-05-14 16:25:38,339][DEBUG][discovery.zen.fd] [ip-10-239-110-94] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] [2013-05-14 16:25:45,943][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) [2013-05-14 16:25:45,978][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) [2013-05-14 16:25:45,998][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) [2013-05-14 16:26:02,795][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) [2013-05-14 16:29:09,448][INFO][cluster.service] [ip-10-239-110-94] removed {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) [2013-05-14 16:29:46,612][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) """ with no entries since then. The log for ip-10-65-49-148 around the time of the start of the CPU burst - """ [2013-05-14 15:18:03,773][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] [2013-05-14 15:18:03,773][INFO][cluster.service] [ip-10-65-49-148] master {new [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}, previous [ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}}, removed {[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) [2013-05-14 15:54:03,982][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}], reason [new cluster stare received and we monitor the wrong master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]] [2013-05-14 16:25:38,340][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] [2013-05-14 16:25:45,944][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) [2013-05-14 16:25:45,979][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) [2013-05-14 16:25:45,998][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) [2013-05-14 16:26:02,797][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) [2013-05-14 16:29:09,452][INFO][cluster.service] [ip-10-65-49-148] removed {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) [2013-05-14 16:29:46,613][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) """ again with no entries since then. I'm not sure how to interpret the logs, but it is possible these nodes thought they were working against two masters (or the wrong master) for a period of time? I'm wondering - - is there anything I can or should do to get this relocation to complete and have the two nodes quiesce their cpu usage? - is it expected for relocation/recovery jobs to take a very long time, or get stuck? - is it expected a cluster can have two masters? - should I file a bug? 馃槈 I'd appreciate any guidance on this, including pointers into ES code. Details - - 0.20.6 - 6 data non master eligible nodes, m1.xlarge, Xms/Xmx 8G, Xmn 400M - 3 master eligible nodes, non data nodes, m1.medium, 2G heaps - minimum master nodes: 2 - ec2 discovery Bill -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bill\_de\_hOra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bill_de_hora/32/2059_2.png) [@Bill\_de\_hOra](https://discuss.elastic.co/u/Bill_de_hOra)\ **Post date:** [May 15, 2013, 11:04am UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/2 "2013-05-15T11:04:32Z")
> relocating state for about 18 hours Which recently quiesced - [https://pbs.twimg.com/media/BKTSpBfCQAAkV0V.png:large](https://pbs.twimg.com/media/BKTSpBfCQAAkV0V.png:large). The relocating node went green in big desk and the other cpu heavy node started shows 2 replicas relocating, but with more relaxed cpu usage. The overall cluster state is green. Again I'm wondering if sustained utilisation of the cpu and very long recovery times are normal/expected behaviour in 0.20.6? (if nothing else, it complicates alarming :). Bill On Wednesday 15 May 2013 at 10:28, Bill de h脫ra wrote: > We experienced a master re-election, seemingly due to a timeout (or underlying throttling of the ec2 discovery mechanism in AWS, we're working through that with Amazon) . Regardless, the previous master got into a bad state similar to [Invalid internal transport message format between 0.20.6 master and client caused re-election 路 Issue #3017 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/3017) (excised from cluster, still reporting green health, needed a restart to join). > > As a result or related to this, I'd has one replica in a relocating state for about 18 hours. I let it run to see if it would complete, but I'm wondering if it will at this point. > > { > "cluster\_name" : "es\_cayovaprod\_0001", > "status" : "yellow", > "timed\_out" : false, > "number\_of\_nodes" : 19, > "number\_of\_data\_nodes" : 6, > "active\_primary\_shards" : 101, > "active\_shards" : 292, > "relocating\_shards" : 1, > "initializing\_shards" : 1, > "unassigned\_shards" : 0 > } > > According to big desk, the cluster state for the replica/shard is - > > ip-10-239-110-94: replica, relocating > ip-10-65-49-148: replica, recovering > > ip-10-36-174-153: primary, green > > ip-10-36-201-76: replica, green > > The index in question is configured to have 2 replicas. > > The first two nodes in transition show a high cpu relative to the other nodes ([https://pbs.twimg.com/media/BKS1BC-CIAE5h\_\_.png:large](https://pbs.twimg.com/media/BKS1BC-CIAE5h__.png:large)), according to htop they're sitting at about 33% which doesn't seem unreasonable in itself (the length of time at that level is the concern) . > > The log for ip-10-239-110-94 around the time of the start of the CPU burst - > > """ > [2013-05-14 15:18:18,219][WARN][indices.cluster] [ip-10-239-110-94] [objects\_0001][22] master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}] marked shard as started, but shard have not been created, mark shard as failed > [2013-05-14 15:18:18,219][WARN][cluster.action.shard] [ip-10-239-110-94] sending failed shard for [objects\_0001][22], node[ybnrwqQ8RPCa69L4SqE7bA], [P], s[STARTED], reason [master [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a} marked shard as started, but shard have not been created, mark shard as failed] > [2013-05-14 15:54:03,981][DEBUG][discovery.zen.fd] [ip-10-239-110-94] [master] restarting fault detection against master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}], reason [new cluster stare received and we monitor the wrong master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]] > [2013-05-14 16:25:38,339][DEBUG][discovery.zen.fd] [ip-10-239-110-94] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] > [2013-05-14 16:25:45,943][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > [2013-05-14 16:25:45,978][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > [2013-05-14 16:25:45,998][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > [2013-05-14 16:26:02,795][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > [2013-05-14 16:29:09,448][INFO][cluster.service] [ip-10-239-110-94] removed {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > [2013-05-14 16:29:46,612][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > """ > > with no entries since then. The log for ip-10-65-49-148 around the time of the start of the CPU burst - > > """ > [2013-05-14 15:18:03,773][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] > [2013-05-14 15:18:03,773][INFO][cluster.service] [ip-10-65-49-148] master {new [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}, previous [ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}}, removed {[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > [2013-05-14 15:54:03,982][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}], reason [new cluster stare received and we monitor the wrong master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]] > [2013-05-14 16:25:38,340][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] > [2013-05-14 16:25:45,944][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > [2013-05-14 16:25:45,979][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > [2013-05-14 16:25:45,998][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > [2013-05-14 16:26:02,797][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > [2013-05-14 16:29:09,452][INFO][cluster.service] [ip-10-65-49-148] removed {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > [2013-05-14 16:29:46,613][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > """ > > again with no entries since then. > > I'm not sure how to interpret the logs, but it is possible these nodes thought they were working against two masters (or the wrong master) for a period of time? > > I'm wondering - > > - is there anything I can or should do to get this relocation to complete and have the two nodes quiesce their cpu usage? > > - is it expected for relocation/recovery jobs to take a very long time, or get stuck? > > - is it expected a cluster can have two masters? > > - should I file a bug? 馃槈 > > I'd appreciate any guidance on this, including pointers into ES code. Details - > > - 0.20.6 > > - 6 data non master eligible nodes, m1.xlarge, Xms/Xmx 8G, Xmn 400M > > - 3 master eligible nodes, non data nodes, m1.medium, 2G heaps > > - minimum master nodes: 2 > > - ec2 discovery > > Bill > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) ([mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [May 15, 2013, 2:11pm UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/3 "2013-05-15T14:11:32Z")
Just out of curiosity did you see a big merge happening just before that? (you might be able to identify quick index growth on disk and then rapid drop) simon On Wednesday, May 15, 2013 1:04:32 PM UTC+2, Bill de h脫ra wrote: > > relocating state for about 18 hours > > Which recently quiesced - > [https://pbs.twimg.com/media/BKTSpBfCQAAkV0V.png:large](https://pbs.twimg.com/media/BKTSpBfCQAAkV0V.png:large). The relocating > node went green in big desk and the other cpu heavy node started shows 2 > replicas relocating, but with more relaxed cpu usage. The overall cluster > state is green. > > Again I'm wondering if sustained utilisation of the cpu and very long > recovery times are normal/expected behaviour in 0.20.6? (if nothing else, > it complicates alarming :). > > Bill > > On Wednesday 15 May 2013 at 10:28, Bill de h脫ra wrote: > > > We experienced a master re-election, seemingly due to a timeout (or > > underlying throttling of the ec2 discovery mechanism in AWS, we're working > > through that with Amazon) . Regardless, the previous master got into a bad > > state similar to > > [Invalid internal transport message format between 0.20.6 master and client caused re-election 路 Issue #3017 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/3017) (excised from > > cluster, still reporting green health, needed a restart to join). > > > > As a result or related to this, I'd has one replica in a relocating > > state for about 18 hours. I let it run to see if it would complete, but I'm > > wondering if it will at this point. > > > > { > > "cluster\_name" : "es\_cayovaprod\_0001", > > "status" : "yellow", > > "timed\_out" : false, > > "number\_of\_nodes" : 19, > > "number\_of\_data\_nodes" : 6, > > "active\_primary\_shards" : 101, > > "active\_shards" : 292, > > "relocating\_shards" : 1, > > "initializing\_shards" : 1, > > "unassigned\_shards" : 0 > > } > > > > According to big desk, the cluster state for the replica/shard is - > > > > ip-10-239-110-94: replica, relocating > > ip-10-65-49-148: replica, recovering > > > > ip-10-36-174-153: primary, green > > > > ip-10-36-201-76: replica, green > > > > The index in question is configured to have 2 replicas. > > > > The first two nodes in transition show a high cpu relative to the other > > nodes ([https://pbs.twimg.com/media/BKS1BC-CIAE5h\_\_.png:large](https://pbs.twimg.com/media/BKS1BC-CIAE5h__.png:large)), according > > to htop they're sitting at about 33% which doesn't seem unreasonable in > > itself (the length of time at that level is the concern) . > > > > The log for ip-10-239-110-94 around the time of the start of the CPU > > burst - > > > > """ > > [2013-05-14 15:18:18,219][WARN][indices.cluster] [ip-10-239-110-94] > > [objects\_0001][22] master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}] marked shard as started, but shard have not > > been created, mark shard as failed > > [2013-05-14 15:18:18,219][WARN][cluster.action.shard] > > [ip-10-239-110-94] sending failed shard for [objects\_0001][22], > > node[ybnrwqQ8RPCa69L4SqE7bA], [P], s[STARTED], reason [master > > [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a} marked shard as started, but shard have not > > been created, mark shard as failed] > > [2013-05-14 15:54:03,981][DEBUG][discovery.zen.fd] [ip-10-239-110-94] > > [master] restarting fault detection against master > > [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c}], reason [new cluster stare received and we > > monitor the wrong master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]] > > [2013-05-14 16:25:38,339][DEBUG][discovery.zen.fd] [ip-10-239-110-94] > > [master] restarting fault detection against master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}], reason [new cluster stare received and we > > monitor the wrong master > > [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c}]] > > [2013-05-14 16:25:45,943][INFO][cluster.service] [ip-10-239-110-94] > > added > > {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, > > data=false},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > [2013-05-14 16:25:45,978][INFO][cluster.service] [ip-10-239-110-94] > > added > > {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, > > data=false},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > [2013-05-14 16:25:45,998][INFO][cluster.service] [ip-10-239-110-94] > > added > > {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, > > data=false},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > [2013-05-14 16:26:02,795][INFO][cluster.service] [ip-10-239-110-94] > > added > > {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > [2013-05-14 16:29:09,448][INFO][cluster.service] [ip-10-239-110-94] > > removed > > {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > [2013-05-14 16:29:46,612][INFO][cluster.service] [ip-10-239-110-94] > > added > > {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > > > """ > > > > with no entries since then. The log for ip-10-65-49-148 around the time > > of the start of the CPU burst - > > > > """ > > [2013-05-14 15:18:03,773][DEBUG][discovery.zen.fd] [ip-10-65-49-148] > > [master] restarting fault detection against master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}], reason [new cluster stare received and we > > monitor the wrong master > > [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c}]] > > [2013-05-14 15:18:03,773][INFO][cluster.service] [ip-10-65-49-148] > > master {new > > [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}, previous > > [ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c}}, removed > > {[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > [2013-05-14 15:54:03,982][DEBUG][discovery.zen.fd] [ip-10-65-49-148] > > [master] restarting fault detection against master > > [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c}], reason [new cluster stare received and we > > monitor the wrong master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]] > > [2013-05-14 16:25:38,340][DEBUG][discovery.zen.fd] [ip-10-65-49-148] > > [master] restarting fault detection against master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}], reason [new cluster stare received and we > > monitor the wrong master > > [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c}]] > > [2013-05-14 16:25:45,944][INFO][cluster.service] [ip-10-65-49-148] > > added > > {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, > > data=false},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > [2013-05-14 16:25:45,979][INFO][cluster.service] [ip-10-65-49-148] > > added > > {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, > > data=false},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > [2013-05-14 16:25:45,998][INFO][cluster.service] [ip-10-65-49-148] > > added > > {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, > > data=false},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > [2013-05-14 16:26:02,797][INFO][cluster.service] [ip-10-65-49-148] > > added > > {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > [2013-05-14 16:29:09,452][INFO][cluster.service] [ip-10-65-49-148] > > removed > > {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > [2013-05-14 16:29:46,613][INFO][cluster.service] [ip-10-65-49-148] > > added > > {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > master=true, zone=eu-west-1a}]) > > > > """ > > > > again with no entries since then. > > > > I'm not sure how to interpret the logs, but it is possible these nodes > > thought they were working against two masters (or the wrong master) for a > > period of time? > > > > I'm wondering - > > > > - is there anything I can or should do to get this relocation to > > complete and have the two nodes quiesce their cpu usage? > > > > - is it expected for relocation/recovery jobs to take a very long time, > > or get stuck? > > > > - is it expected a cluster can have two masters? > > > > - should I file a bug? 馃槈 > > > > I'd appreciate any guidance on this, including pointers into ES code. > > Details - > > > > - 0.20.6 > > > > - 6 data non master eligible nodes, m1.xlarge, Xms/Xmx 8G, Xmn 400M > > > > - 3 master eligible nodes, non data nodes, m1.medium, 2G heaps > > > > - minimum master nodes: 2 > > > > - ec2 discovery > > > > Bill > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \ (mailto: > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bill\_de\_hOra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bill_de_hora/32/2059_2.png) [@Bill\_de\_hOra](https://discuss.elastic.co/u/Bill_de_hOra)\ **Post date:** [May 15, 2013, 4:47pm UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/4 "2013-05-15T16:47:29Z")
> Just out of curiosity did you see a big merge happening just before that? Unsure about merge, but I can see a disk write spike a few hours before the cpu levelled out on _all_ data nodes. What happened since then is interesting - > > > ip-10-239-110-94: replica, relocating - This node became empty according to bigdesk, leaving 5 of 6 carrying all the data. - At that point I decided a shutdown/restart was in order for ip-10-239-110-94 - The cluster it seems was starting to move data back to ip-10-239-110-94 according to bigdesk (I had executed shutdown at that point, but not restart, went ahead and performed the restart) - Post restart the ip-10-239-110-94 node has just 2 replicas (out of over 300 files) - The ip-10-239-70-202 node is still relocating two replicas. Those are targeted for ip-10-239-110-94 (they are different to the ones on the ip-10-239-110-94 node) I'm wondering if cluster.routing.allocation.\* settings has something to do with the limit on relocations that are allowed, but regardless, it's taking a long time for them to finish. Some of these files are just kB in size (it's a young index). Bill On Wednesday 15 May 2013 at 15:11, simonw wrote: > Just out of curiosity did you see a big merge happening just before that? (you might be able to identify quick index growth on disk and then rapid drop) > > simon > > On Wednesday, May 15, 2013 1:04:32 PM UTC+2, Bill de h脫ra wrote: > > > > relocating state for about 18 hours > > > > Which recently quiesced - [https://pbs.twimg.com/media/BKTSpBfCQAAkV0V.png:large](https://pbs.twimg.com/media/BKTSpBfCQAAkV0V.png:large). The relocating node went green in big desk and the other cpu heavy node started shows 2 replicas relocating, but with more relaxed cpu usage. The overall cluster state is green. > > > > Again I'm wondering if sustained utilisation of the cpu and very long recovery times are normal/expected behaviour in 0.20.6? (if nothing else, it complicates alarming :). > > > > Bill > > > > On Wednesday 15 May 2013 at 10:28, Bill de h脫ra wrote: > > > > > We experienced a master re-election, seemingly due to a timeout (or underlying throttling of the ec2 discovery mechanism in AWS, we're working through that with Amazon) . Regardless, the previous master got into a bad state similar to [Invalid internal transport message format between 0.20.6 master and client caused re-election 路 Issue #3017 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/3017) (excised from cluster, still reporting green health, needed a restart to join). > > > > > > As a result or related to this, I'd has one replica in a relocating state for about 18 hours. I let it run to see if it would complete, but I'm wondering if it will at this point. > > > > > > { > > > "cluster\_name" : "es\_cayovaprod\_0001", > > > "status" : "yellow", > > > "timed\_out" : false, > > > "number\_of\_nodes" : 19, > > > "number\_of\_data\_nodes" : 6, > > > "active\_primary\_shards" : 101, > > > "active\_shards" : 292, > > > "relocating\_shards" : 1, > > > "initializing\_shards" : 1, > > > "unassigned\_shards" : 0 > > > } > > > > > > According to big desk, the cluster state for the replica/shard is - > > > > > > ip-10-239-110-94: replica, relocating > > > ip-10-65-49-148: replica, recovering > > > > > > ip-10-36-174-153: primary, green > > > > > > ip-10-36-201-76: replica, green > > > > > > The index in question is configured to have 2 replicas. > > > > > > The first two nodes in transition show a high cpu relative to the other nodes ([https://pbs.twimg.com/media/BKS1BC-CIAE5h\_\_.png:large](https://pbs.twimg.com/media/BKS1BC-CIAE5h__.png:large)), according to htop they're sitting at about 33% which doesn't seem unreasonable in itself (the length of time at that level is the concern) . > > > > > > The log for ip-10-239-110-94 around the time of the start of the CPU burst - > > > > > > """ > > > [2013-05-14 15:18:18,219][WARN][indices.cluster] [ip-10-239-110-94] [objects\_0001][22] master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}] marked shard as started, but shard have not been created, mark shard as failed > > > [2013-05-14 15:18:18,219][WARN][cluster.action.shard] [ip-10-239-110-94] sending failed shard for [objects\_0001][22], node[ybnrwqQ8RPCa69L4SqE7bA], [P], s[STARTED], reason [master [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a} marked shard as started, but shard have not been created, mark shard as failed] > > > [2013-05-14 15:54:03,981][DEBUG][discovery.zen.fd] [ip-10-239-110-94] [master] restarting fault detection against master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}], reason [new cluster stare received and we monitor the wrong master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]] > > > [2013-05-14 16:25:38,339][DEBUG][discovery.zen.fd] [ip-10-239-110-94] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] > > > [2013-05-14 16:25:45,943][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > [2013-05-14 16:25:45,978][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > [2013-05-14 16:25:45,998][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > [2013-05-14 16:26:02,795][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > [2013-05-14 16:29:09,448][INFO][cluster.service] [ip-10-239-110-94] removed {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > [2013-05-14 16:29:46,612][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > > """ > > > > > > with no entries since then. The log for ip-10-65-49-148 around the time of the start of the CPU burst - > > > > > > """ > > > [2013-05-14 15:18:03,773][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] > > > [2013-05-14 15:18:03,773][INFO][cluster.service] [ip-10-65-49-148] master {new [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}, previous [ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}}, removed {[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > [2013-05-14 15:54:03,982][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}], reason [new cluster stare received and we monitor the wrong master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]] > > > [2013-05-14 16:25:38,340][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] > > > [2013-05-14 16:25:45,944][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > [2013-05-14 16:25:45,979][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > [2013-05-14 16:25:45,998][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > [2013-05-14 16:26:02,797][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > [2013-05-14 16:29:09,452][INFO][cluster.service] [ip-10-65-49-148] removed {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > [2013-05-14 16:29:46,613][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > > """ > > > > > > again with no entries since then. > > > > > > I'm not sure how to interpret the logs, but it is possible these nodes thought they were working against two masters (or the wrong master) for a period of time? > > > > > > I'm wondering - > > > > > > - is there anything I can or should do to get this relocation to complete and have the two nodes quiesce their cpu usage? > > > > > > - is it expected for relocation/recovery jobs to take a very long time, or get stuck? > > > > > > - is it expected a cluster can have two masters? > > > > > > - should I file a bug? 馃槈 > > > > > > I'd appreciate any guidance on this, including pointers into ES code. Details - > > > > > > - 0.20.6 > > > > > > - 6 data non master eligible nodes, m1.xlarge, Xms/Xmx 8G, Xmn 400M > > > > > > - 3 master eligible nodes, non data nodes, m1.medium, 2G heaps > > > > > > - minimum master nodes: 2 > > > > > > - ec2 discovery > > > > > > Bill > > > > > > -- > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) (javascript:) ([mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (javascript:)). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) ([mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [May 16, 2013, 9:00am UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/5 "2013-05-16T09:00:06Z")
Hi Bill Is there anything happening in the logs? I've seen instances on EC2 with bad disks / degraded hardware which result in IO errors preventing replication clint On 15 May 2013 18:47, Bill de h脫ra [bill@dehora.net](mailto:bill@dehora.net) wrote: > > Just out of curiosity did you see a big merge happening just before that? > > Unsure about merge, but I can see a disk write spike a few hours before > the cpu levelled out on _all_ data nodes. > > What happened since then is interesting - > > > > > ip-10-239-110-94: replica, relocating > > - This node became empty according to bigdesk, leaving 5 of 6 carrying > all the data. > > - At that point I decided a shutdown/restart was in order for > ip-10-239-110-94 > > - The cluster it seems was starting to move data back to ip-10-239-110-94 > according to bigdesk (I had executed shutdown at that point, but not > restart, went ahead and performed the restart) > > - Post restart the ip-10-239-110-94 node has just 2 replicas (out of over > 300 files) > > - The ip-10-239-70-202 node is still relocating two replicas. Those are > targeted for ip-10-239-110-94 (they are different to the ones on the > ip-10-239-110-94 node) > > I'm wondering if cluster.routing.allocation.\* settings has something to do > with the limit on relocations that are allowed, but regardless, it's taking > a long time for them to finish. Some of these files are just kB in size > (it's a young index). > > Bill > > On Wednesday 15 May 2013 at 15:11, simonw wrote: > > > Just out of curiosity did you see a big merge happening just before > > that? (you might be able to identify quick index growth on disk and then > > rapid drop) > > > > simon > > > > On Wednesday, May 15, 2013 1:04:32 PM UTC+2, Bill de h脫ra wrote: > > > > > > relocating state for about 18 hours > > > > > > Which recently quiesced - > > > [https://pbs.twimg.com/media/BKTSpBfCQAAkV0V.png:large](https://pbs.twimg.com/media/BKTSpBfCQAAkV0V.png:large). The relocating > > > node went green in big desk and the other cpu heavy node started shows 2 > > > replicas relocating, but with more relaxed cpu usage. The overall cluster > > > state is green. > > > > > > Again I'm wondering if sustained utilisation of the cpu and very long > > > recovery times are normal/expected behaviour in 0.20.6? (if nothing else, > > > it complicates alarming :). > > > > > > Bill > > > > > > On Wednesday 15 May 2013 at 10:28, Bill de h脫ra wrote: > > > > > > > We experienced a master re-election, seemingly due to a timeout (or > > > > underlying throttling of the ec2 discovery mechanism in AWS, we're working > > > > through that with Amazon) . Regardless, the previous master got into a bad > > > > state similar to > > > > [Invalid internal transport message format between 0.20.6 master and client caused re-election 路 Issue #3017 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/3017) (excised from > > > > cluster, still reporting green health, needed a restart to join). > > > > > > > > As a result or related to this, I'd has one replica in a relocating > > > > state for about 18 hours. I let it run to see if it would complete, but I'm > > > > wondering if it will at this point. > > > > > > > > { > > > > "cluster\_name" : "es\_cayovaprod\_0001", > > > > "status" : "yellow", > > > > "timed\_out" : false, > > > > "number\_of\_nodes" : 19, > > > > "number\_of\_data\_nodes" : 6, > > > > "active\_primary\_shards" : 101, > > > > "active\_shards" : 292, > > > > "relocating\_shards" : 1, > > > > "initializing\_shards" : 1, > > > > "unassigned\_shards" : 0 > > > > } > > > > > > > > According to big desk, the cluster state for the replica/shard is - > > > > > > > > ip-10-239-110-94: replica, relocating > > > > ip-10-65-49-148: replica, recovering > > > > > > > > ip-10-36-174-153: primary, green > > > > > > > > ip-10-36-201-76: replica, green > > > > > > > > The index in question is configured to have 2 replicas. > > > > > > > > The first two nodes in transition show a high cpu relative to the > > > > other nodes ([https://pbs.twimg.com/media/BKS1BC-CIAE5h\_\_.png:large](https://pbs.twimg.com/media/BKS1BC-CIAE5h__.png:large)), > > > > according to htop they're sitting at about 33% which doesn't seem > > > > unreasonable in itself (the length of time at that level is the concern) . > > > > > > > > The log for ip-10-239-110-94 around the time of the start of the CPU > > > > burst - > > > > > > > > """ > > > > [2013-05-14 15:18:18,219][WARN][indices.cluster] > > > > [ip-10-239-110-94] [objects\_0001][22] master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}] marked shard as started, but shard have not > > > > been created, mark shard as failed > > > > [2013-05-14 15:18:18,219][WARN][cluster.action.shard] > > > > [ip-10-239-110-94] sending failed shard for [objects\_0001][22], > > > > node[ybnrwqQ8RPCa69L4SqE7bA], [P], s[STARTED], reason [master > > > > [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a} marked shard as started, but shard have not > > > > been created, mark shard as failed] > > > > [2013-05-14 15:54:03,981][DEBUG][discovery.zen.fd] > > > > [ip-10-239-110-94] [master] restarting fault detection against master > > > > [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c}], reason [new cluster stare received and we > > > > monitor the wrong master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]] > > > > [2013-05-14 16:25:38,339][DEBUG][discovery.zen.fd] > > > > [ip-10-239-110-94] [master] restarting fault detection against master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}], reason [new cluster stare received and we > > > > monitor the wrong master > > > > [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c}]] > > > > [2013-05-14 16:25:45,943][INFO][cluster.service] > > > > [ip-10-239-110-94] added > > > > {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, > > > > data=false},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > [2013-05-14 16:25:45,978][INFO][cluster.service] > > > > [ip-10-239-110-94] added > > > > {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, > > > > data=false},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > [2013-05-14 16:25:45,998][INFO][cluster.service] > > > > [ip-10-239-110-94] added > > > > {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, > > > > data=false},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > [2013-05-14 16:26:02,795][INFO][cluster.service] > > > > [ip-10-239-110-94] added > > > > {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > [2013-05-14 16:29:09,448][INFO][cluster.service] > > > > [ip-10-239-110-94] removed > > > > {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > [2013-05-14 16:29:46,612][INFO][cluster.service] > > > > [ip-10-239-110-94] added > > > > {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > > > > > """ > > > > > > > > with no entries since then. The log for ip-10-65-49-148 around the > > > > time of the start of the CPU burst - > > > > > > > > """ > > > > [2013-05-14 15:18:03,773][DEBUG][discovery.zen.fd] > > > > [ip-10-65-49-148] [master] restarting fault detection against master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}], reason [new cluster stare received and we > > > > monitor the wrong master > > > > [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c}]] > > > > [2013-05-14 15:18:03,773][INFO][cluster.service] [ip-10-65-49-148] > > > > master {new > > > > [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}, previous > > > > [ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c}}, removed > > > > {[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > [2013-05-14 15:54:03,982][DEBUG][discovery.zen.fd] > > > > [ip-10-65-49-148] [master] restarting fault detection against master > > > > [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c}], reason [new cluster stare received and we > > > > monitor the wrong master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]] > > > > [2013-05-14 16:25:38,340][DEBUG][discovery.zen.fd] > > > > [ip-10-65-49-148] [master] restarting fault detection against master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}], reason [new cluster stare received and we > > > > monitor the wrong master > > > > [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c}]] > > > > [2013-05-14 16:25:45,944][INFO][cluster.service] [ip-10-65-49-148] > > > > added {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, > > > > data=false},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > [2013-05-14 16:25:45,979][INFO][cluster.service] [ip-10-65-49-148] > > > > added {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, > > > > data=false},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > [2013-05-14 16:25:45,998][INFO][cluster.service] [ip-10-65-49-148] > > > > added {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, > > > > data=false},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > [2013-05-14 16:26:02,797][INFO][cluster.service] [ip-10-65-49-148] > > > > added {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > [2013-05-14 16:29:09,452][INFO][cluster.service] [ip-10-65-49-148] > > > > removed {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > [2013-05-14 16:29:46,613][INFO][cluster.service] [ip-10-65-49-148] > > > > added {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, > > > > master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master > > > > [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, > > > > master=true, zone=eu-west-1a}]) > > > > > > > > """ > > > > > > > > again with no entries since then. > > > > > > > > I'm not sure how to interpret the logs, but it is possible these > > > > nodes thought they were working against two masters (or the wrong master) > > > > for a period of time? > > > > > > > > I'm wondering - > > > > > > > > - is there anything I can or should do to get this relocation to > > > > complete and have the two nodes quiesce their cpu usage? > > > > > > > > - is it expected for relocation/recovery jobs to take a very long > > > > time, or get stuck? > > > > > > > > - is it expected a cluster can have two masters? > > > > > > > > - should I file a bug? 馃槈 > > > > > > > > I'd appreciate any guidance on this, including pointers into ES > > > > code. Details - > > > > > > > > - 0.20.6 > > > > > > > > - 6 data non master eligible nodes, m1.xlarge, Xms/Xmx 8G, Xmn 400M > > > > > > > > - 3 master eligible nodes, non data nodes, m1.medium, 2G heaps > > > > > > > > - minimum master nodes: 2 > > > > > > > > - ec2 discovery > > > > > > > > Bill > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) (javascript:) (mailto: > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (javascript:)). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto: > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bill\_de\_hOra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bill_de_hora/32/2059_2.png) [@Bill\_de\_hOra](https://discuss.elastic.co/u/Bill_de_hOra)\ **Post date:** [May 16, 2013, 1:15pm UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/6 "2013-05-16T13:15:38Z")
Hi Clinton, on the node with 2 pending relocations, last logs line it [2013-05-15 14:06:45,688][DEBUG][discovery.ec2] [ip-10-239-70-202] got a new state from master node, though we are already trying to rejoin the cluster Otherwise it seems fine, the disk is ok, nothing obvious in the logs. Unclear whether I should restart this node, or do anything at all, given the cluster is green, but allocation is uneven. The ip-10-239-110-94 still only has 2 replicas (compared to 87 on this one) Bill On Thursday 16 May 2013 at 10:00, Clinton Gormley wrote: > Hi Bill > > Is there anything happening in the logs? > > I've seen instances on EC2 with bad disks / degraded hardware which result in IO errors preventing replication > > clint > > On 15 May 2013 18:47, Bill de h脫ra \<[bill@dehora.net](mailto:bill@dehora.net) ([mailto:bill@dehora.net](mailto:bill@dehora.net))\> wrote: > > > > Just out of curiosity did you see a big merge happening just before that? > > > > Unsure about merge, but I can see a disk write spike a few hours before the cpu levelled out on _all_ data nodes. > > > > What happened since then is interesting - > > > > > > > ip-10-239-110-94: replica, relocating > > > > - This node became empty according to bigdesk, leaving 5 of 6 carrying all the data. > > > > - At that point I decided a shutdown/restart was in order for ip-10-239-110-94 > > > > - The cluster it seems was starting to move data back to ip-10-239-110-94 according to bigdesk (I had executed shutdown at that point, but not restart, went ahead and performed the restart) > > > > - Post restart the ip-10-239-110-94 node has just 2 replicas (out of over 300 files) > > > > - The ip-10-239-70-202 node is still relocating two replicas. Those are targeted for ip-10-239-110-94 (they are different to the ones on the ip-10-239-110-94 node) > > > > I'm wondering if cluster.routing.allocation.\* settings has something to do with the limit on relocations that are allowed, but regardless, it's taking a long time for them to finish. Some of these files are just kB in size (it's a young index). > > > > Bill > > > > On Wednesday 15 May 2013 at 15:11, simonw wrote: > > > > > Just out of curiosity did you see a big merge happening just before that? (you might be able to identify quick index growth on disk and then rapid drop) > > > > > > simon > > > > > > On Wednesday, May 15, 2013 1:04:32 PM UTC+2, Bill de h脫ra wrote: > > > > > > > > relocating state for about 18 hours > > > > > > > > Which recently quiesced - [https://pbs.twimg.com/media/BKTSpBfCQAAkV0V.png:large](https://pbs.twimg.com/media/BKTSpBfCQAAkV0V.png:large). The relocating node went green in big desk and the other cpu heavy node started shows 2 replicas relocating, but with more relaxed cpu usage. The overall cluster state is green. > > > > > > > > Again I'm wondering if sustained utilisation of the cpu and very long recovery times are normal/expected behaviour in 0.20.6? (if nothing else, it complicates alarming :). > > > > > > > > Bill > > > > > > > > On Wednesday 15 May 2013 at 10:28, Bill de h脫ra wrote: > > > > > > > > > We experienced a master re-election, seemingly due to a timeout (or underlying throttling of the ec2 discovery mechanism in AWS, we're working through that with Amazon) . Regardless, the previous master got into a bad state similar to [Invalid internal transport message format between 0.20.6 master and client caused re-election 路 Issue #3017 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/3017) (excised from cluster, still reporting green health, needed a restart to join). > > > > > > > > > > As a result or related to this, I'd has one replica in a relocating state for about 18 hours. I let it run to see if it would complete, but I'm wondering if it will at this point. > > > > > > > > > > { > > > > > "cluster\_name" : "es\_cayovaprod\_0001", > > > > > "status" : "yellow", > > > > > "timed\_out" : false, > > > > > "number\_of\_nodes" : 19, > > > > > "number\_of\_data\_nodes" : 6, > > > > > "active\_primary\_shards" : 101, > > > > > "active\_shards" : 292, > > > > > "relocating\_shards" : 1, > > > > > "initializing\_shards" : 1, > > > > > "unassigned\_shards" : 0 > > > > > } > > > > > > > > > > According to big desk, the cluster state for the replica/shard is - > > > > > > > > > > ip-10-239-110-94: replica, relocating > > > > > ip-10-65-49-148: replica, recovering > > > > > > > > > > ip-10-36-174-153: primary, green > > > > > > > > > > ip-10-36-201-76: replica, green > > > > > > > > > > The index in question is configured to have 2 replicas. > > > > > > > > > > The first two nodes in transition show a high cpu relative to the other nodes ([https://pbs.twimg.com/media/BKS1BC-CIAE5h\_\_.png:large](https://pbs.twimg.com/media/BKS1BC-CIAE5h__.png:large)), according to htop they're sitting at about 33% which doesn't seem unreasonable in itself (the length of time at that level is the concern) . > > > > > > > > > > The log for ip-10-239-110-94 around the time of the start of the CPU burst - > > > > > > > > > > """ > > > > > [2013-05-14 15:18:18,219][WARN][indices.cluster] [ip-10-239-110-94] [objects\_0001][22] master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}] marked shard as started, but shard have not been created, mark shard as failed > > > > > [2013-05-14 15:18:18,219][WARN][cluster.action.shard] [ip-10-239-110-94] sending failed shard for [objects\_0001][22], node[ybnrwqQ8RPCa69L4SqE7bA], [P], s[STARTED], reason [master [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a} marked shard as started, but shard have not been created, mark shard as failed] > > > > > [2013-05-14 15:54:03,981][DEBUG][discovery.zen.fd] [ip-10-239-110-94] [master] restarting fault detection against master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}], reason [new cluster stare received and we monitor the wrong master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]] > > > > > [2013-05-14 16:25:38,339][DEBUG][discovery.zen.fd] [ip-10-239-110-94] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] > > > > > [2013-05-14 16:25:45,943][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > [2013-05-14 16:25:45,978][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > [2013-05-14 16:25:45,998][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > [2013-05-14 16:26:02,795][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > [2013-05-14 16:29:09,448][INFO][cluster.service] [ip-10-239-110-94] removed {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > [2013-05-14 16:29:46,612][INFO][cluster.service] [ip-10-239-110-94] added {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > > > > > > """ > > > > > > > > > > with no entries since then. The log for ip-10-65-49-148 around the time of the start of the CPU burst - > > > > > > > > > > """ > > > > > [2013-05-14 15:18:03,773][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] > > > > > [2013-05-14 15:18:03,773][INFO][cluster.service] [ip-10-65-49-148] master {new [ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}, previous [ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}}, removed {[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > [2013-05-14 15:54:03,982][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}], reason [new cluster stare received and we monitor the wrong master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]] > > > > > [2013-05-14 16:25:38,340][DEBUG][discovery.zen.fd] [ip-10-65-49-148] [master] restarting fault detection against master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}], reason [new cluster stare received and we monitor the wrong master [[ip-10-34-144-149][IsP0kjtRS6KJ-9R3hZehwQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c}]] > > > > > [2013-05-14 16:25:45,944][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-139-232][COIAyFiiRumWsc1h\_c1uhQ][inet[/10.36.139.232:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > [2013-05-14 16:25:45,979][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-129-154][an52EiHdTaiFBvaxzq-KOQ][inet[/10.36.129.154:9302]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > [2013-05-14 16:25:45,998][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-36-129-154][VZzB17qkTneJNJLZizFtcA][inet[/10.36.129.154:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > [2013-05-14 16:26:02,797][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > [2013-05-14 16:29:09,452][INFO][cluster.service] [ip-10-65-49-148] removed {[ip-10-34-144-149][6EbGNFdOQv6\_ny2\_jdPQyg][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > [2013-05-14 16:29:46,613][INFO][cluster.service] [ip-10-65-49-148] added {[ip-10-34-144-149][UM03BjRfS3OmTu9QLcWneQ][inet[/10.34.144.149:9300]]{data=false, master=true, zone=eu-west-1c},}, reason: zen-disco-receive(from master [[ip-10-36-171-62][j-Gbp\_DDSHqYk3e25MPf4Q][inet[/10.36.171.62:9300]]{data=false, master=true, zone=eu-west-1a}]) > > > > > > > > > > """ > > > > > > > > > > again with no entries since then. > > > > > > > > > > I'm not sure how to interpret the logs, but it is possible these nodes thought they were working against two masters (or the wrong master) for a period of time? > > > > > > > > > > I'm wondering - > > > > > > > > > > - is there anything I can or should do to get this relocation to complete and have the two nodes quiesce their cpu usage? > > > > > > > > > > - is it expected for relocation/recovery jobs to take a very long time, or get stuck? > > > > > > > > > > - is it expected a cluster can have two masters? > > > > > > > > > > - should I file a bug? 馃槈 > > > > > > > > > > I'd appreciate any guidance on this, including pointers into ES code. Details - > > > > > > > > > > - 0.20.6 > > > > > > > > > > - 6 data non master eligible nodes, m1.xlarge, Xms/Xmx 8G, Xmn 400M > > > > > > > > > > - 3 master eligible nodes, non data nodes, m1.medium, 2G heaps > > > > > > > > > > - minimum master nodes: 2 > > > > > > > > > > - ec2 discovery > > > > > > > > > > Bill > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) ([mailto:elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)) (javascript:) ([mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto:elasticsearch%2Bunsubscribe@googlegroups.com) (javascript:)). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto:elasticsearch%2Bunsubscribe@googlegroups.com) ([mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto:elasticsearch%2Bunsubscribe@googlegroups.com)). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto:elasticsearch%2Bunsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) ([mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [May 16, 2013, 1:23pm UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/7 "2013-05-16T13:23:08Z")
On 16 May 2013 15:15, Bill de h脫ra [bill@dehora.net](mailto:bill@dehora.net) wrote: > Otherwise it seems fine, the disk is ok, nothing obvious in the logs. > Unclear whether I should restart this node, or do anything at all, given > the cluster is green, but allocation is uneven. The ip-10-239-110-94 still > only has 2 replicas (compared to 87 on this one) How many shards in total on each node (ie primaries and replicas) clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bill\_de\_hOra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bill_de_hora/32/2059_2.png) [@Bill\_de\_hOra](https://discuss.elastic.co/u/Bill_de_hOra)\ **Post date:** [May 16, 2013, 7:52pm UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/8 "2013-05-16T19:52:25Z")
> How many shards in total on each node (ie primaries and replicas) 293 in total, they are organised into zones zone a ip-10-36-174-153: 48 ip-10-36-201-76: 49 zone b ip-10-251-55-42: 48 ip-10-65-49-148: 52 zone c ip-10-239-110-94: 2 ip-10-239-70-202: 94 Bill On Thursday 16 May 2013 at 14:23, Clinton Gormley wrote: > On 16 May 2013 15:15, Bill de h脫ra \<[bill@dehora.net](mailto:bill@dehora.net) ([mailto:bill@dehora.net](mailto:bill@dehora.net))\> wrote: > > > Otherwise it seems fine, the disk is ok, nothing obvious in the logs. Unclear whether I should restart this node, or do anything at all, given the cluster is green, but allocation is uneven. The ip-10-239-110-94 still only has 2 replicas (compared to 87 on this one) > > How many shards in total on each node (ie primaries and replicas) > > clint > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) ([mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [May 17, 2013, 10:49am UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/9 "2013-05-17T10:49:55Z")
Hmmm, something definitely looks wrong. The cluster.routing.allocation settings may be the cause. Could you provide details of those? Also, I'd consider dropping that instance and creating a new one, just in case. See if that fixes the issue clint On 16 May 2013 21:52, Bill de h脫ra [bill@dehora.net](mailto:bill@dehora.net) wrote: > > How many shards in total on each node (ie primaries and replicas) > > 293 in total, they are organised into zones > > zone a > ip-10-36-174-153: 48 > ip-10-36-201-76: 49 > > zone b > ip-10-251-55-42: 48 > > ip-10-65-49-148: 52 > > zone c > ip-10-239-110-94: 2 > ip-10-239-70-202: 94 > > Bill > > On Thursday 16 May 2013 at 14:23, Clinton Gormley wrote: > > > On 16 May 2013 15:15, Bill de h脫ra \<[bill@dehora.net](mailto:bill@dehora.net) (mailto: > > [bill@dehora.net](mailto:bill@dehora.net))\> wrote: > > > > > Otherwise it seems fine, the disk is ok, nothing obvious in the logs. > > > Unclear whether I should restart this node, or do anything at all, given > > > the cluster is green, but allocation is uneven. The ip-10-239-110-94 still > > > only has 2 replicas (compared to 87 on this one) > > > > How many shards in total on each node (ie primaries and replicas) > > > > clint > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto: > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bill\_de\_hOra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bill_de_hora/32/2059_2.png) [@Bill\_de\_hOra](https://discuss.elastic.co/u/Bill_de_hOra)\ **Post date:** [May 17, 2013, 11:21am UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/10 "2013-05-17T11:21:31Z")
> The cluster.routing.allocation settings may be the cause. Could you provide details of those? Here you go as passed in via -Des.config - """ node.zone: eu-west-1a # eu-west-1b, eu-west-1c cluster.routing.allocation.awareness.attributes: zone """ Otherwise, using defaults from 0.20.6. Running 'curl -XGET localhost:9200/\_cluster/settings' returns an empty block {"persistent":{},"transient":{}} Bill On Friday 17 May 2013 at 11:49, Clinton Gormley wrote: > Hmmm, something definitely looks wrong. The cluster.routing.allocation settings may be the cause. Could you provide details of those? > > Also, I'd consider dropping that instance and creating a new one, just in case. See if that fixes the issue > > clint > > On 16 May 2013 21:52, Bill de h脫ra \<[bill@dehora.net](mailto:bill@dehora.net) ([mailto:bill@dehora.net](mailto:bill@dehora.net))\> wrote: > > > > How many shards in total on each node (ie primaries and replicas) > > > > 293 in total, they are organised into zones > > > > zone a > > ip-10-36-174-153: 48 > > ip-10-36-201-76: 49 > > > > zone b > > ip-10-251-55-42: 48 > > > > ip-10-65-49-148: 52 > > > > zone c > > ip-10-239-110-94: 2 > > ip-10-239-70-202: 94 > > > > Bill > > > > On Thursday 16 May 2013 at 14:23, Clinton Gormley wrote: > > > > > On 16 May 2013 15:15, Bill de h脫ra \<[bill@dehora.net](mailto:bill@dehora.net) ([mailto:bill@dehora.net](mailto:bill@dehora.net)) ([mailto:bill@dehora.net](mailto:bill@dehora.net))\> wrote: > > > > > > > Otherwise it seems fine, the disk is ok, nothing obvious in the logs. Unclear whether I should restart this node, or do anything at all, given the cluster is green, but allocation is uneven. The ip-10-239-110-94 still only has 2 replicas (compared to 87 on this one) > > > > > > How many shards in total on each node (ie primaries and replicas) > > > > > > clint > > > > > > -- > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto:elasticsearch%2Bunsubscribe@googlegroups.com) ([mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto:elasticsearch%2Bunsubscribe@googlegroups.com)). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto:elasticsearch%2Bunsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) ([mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [May 17, 2013, 11:30am UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/11 "2013-05-17T11:30:27Z")
Hmm... If it is trying to relocate two shards, and they are stuck, then something is wrong. You want to look in the logs of the source and destination nodes and find what the problem is. Also, as per my prev email, I'd suggest blowing away that instance and creating a new one, to see if that fixes it. Make sure that the problem node actually has a 'zone' attribute set - if it doesn't then it won't have shards assigned to it clint On 17 May 2013 13:21, Bill de h脫ra [bill@dehora.net](mailto:bill@dehora.net) wrote: > > The cluster.routing.allocation settings may be the cause. Could you > > provide details of those? > > Here you go as passed in via -Des.config - > > """ > node.zone: eu-west-1a # eu-west-1b, eu-west-1c > cluster.routing.allocation.awareness.attributes: zone > > """ > > Otherwise, using defaults from 0.20.6. Running 'curl -XGET > localhost:9200/\_cluster/settings' returns an empty block > > {"persistent":{},"transient":{}} > > Bill > > On Friday 17 May 2013 at 11:49, Clinton Gormley wrote: > > > Hmmm, something definitely looks wrong. The cluster.routing.allocation > > settings may be the cause. Could you provide details of those? > > > > Also, I'd consider dropping that instance and creating a new one, just > > in case. See if that fixes the issue > > > > clint > > > > On 16 May 2013 21:52, Bill de h脫ra \<[bill@dehora.net](mailto:bill@dehora.net) (mailto: > > [bill@dehora.net](mailto:bill@dehora.net))\> wrote: > > > > > > How many shards in total on each node (ie primaries and replicas) > > > > > > 293 in total, they are organised into zones > > > > > > zone a > > > ip-10-36-174-153: 48 > > > ip-10-36-201-76: 49 > > > > > > zone b > > > ip-10-251-55-42: 48 > > > > > > ip-10-65-49-148: 52 > > > > > > zone c > > > ip-10-239-110-94: 2 > > > ip-10-239-70-202: 94 > > > > > > Bill > > > > > > On Thursday 16 May 2013 at 14:23, Clinton Gormley wrote: > > > > > > > On 16 May 2013 15:15, Bill de h脫ra \<[bill@dehora.net](mailto:bill@dehora.net) (mailto: > > > > [bill@dehora.net](mailto:bill@dehora.net)) ([mailto:bill@dehora.net](mailto:bill@dehora.net))\> wrote: > > > > > > > > > Otherwise it seems fine, the disk is ok, nothing obvious in the > > > > > logs. Unclear whether I should restart this node, or do anything at all, > > > > > given the cluster is green, but allocation is uneven. The ip-10-239-110-94 > > > > > still only has 2 replicas (compared to 87 on this one) > > > > > > > > How many shards in total on each node (ie primaries and replicas) > > > > > > > > clint > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto: > > > > elasticsearch%2Bunsubscribe@googlegroups.com) (mailto: > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto: > > > > elasticsearch%2Bunsubscribe@googlegroups.com)). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto: > > > elasticsearch%2Bunsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) (mailto: > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:36am UTC](https://discuss.elastic.co/t/replica-stuck-in-relocation-high-cpu-usage/11970/12 "2017-07-06T02:36:05Z")