### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [October 2, 2012, 5:19pm UTC](https://discuss.elastic.co/t/shards-fail-to-reallocate/9212/1 "2012-10-02T17:19:54Z")
I have a 12 node cluster running 0.19.8 with two-three 100gb indices that
have between six-eight shards and one replica. Not in production, so there
are not many queries. One index gets bulk updated about every 2 hours.
One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been giving
us troubles, accepting connections but not processing them. Occasionaly
dumps large 10GB+ heapdumps.
After the last restart of that node (reallocation still enabled), two nodes
attempt to move shards to it, but they stall part way. There has been no
progress in the past day and the restarted node still contains no active
shards.
The gist provides the cluster stats, node stats, and the jstack of the
three servers involved in the reallocation.
>
>
> There are more than three files. show original
Cheers,
Ivan
--
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [October 2, 2012, 5:32pm UTC](https://discuss.elastic.co/t/shards-fail-to-reallocate/9212/2 "2012-10-02T17:32:34Z")
I should add that the index with the shard issue is not being queried
against and not receiving any updates. It is merely an older version of the
current index.
The overall issue is not why these shards are not reallocating, but why is
this node misbehaving?
On Tue, Oct 2, 2012 at 10:19 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> I have a 12 node cluster running 0.19.8 with two-three 100gb indices that
> have between six-eight shards and one replica. Not in production, so there
> are not many queries. One index gets bulk updated about every 2 hours.
>
> One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been
> giving us troubles, accepting connections but not processing them.
> Occasionaly dumps large 10GB+ heapdumps.
>
> After the last restart of that node (reallocation still enabled), two
> nodes attempt to move shards to it, but they stall part way. There has been
> no progress in the past day and the restarted node still contains no active
> shards.
>
> The gist provides the cluster stats, node stats, and the jstack of the
> three servers involved in the reallocation.
>
> [Reallocation failure · GitHub](https://gist.github.com/fb44ff1122fbf293ba5d)
>
> Cheers,
>
> Ivan
--
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [October 2, 2012, 6:25pm UTC](https://discuss.elastic.co/t/shards-fail-to-reallocate/9212/3 "2012-10-02T18:25:07Z")
I did not understand the misbehavior part, do you mean that shards fail to relocate to it? Or the fact that it has memory problems?
On Oct 2, 2012, at 1:32 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> I should add that the index with the shard issue is not being queried against and not receiving any updates. It is merely an older version of the current index.
>
> The overall issue is not why these shards are not reallocating, but why is this node misbehaving?
>
> On Tue, Oct 2, 2012 at 10:19 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> I have a 12 node cluster running 0.19.8 with two-three 100gb indices that have between six-eight shards and one replica. Not in production, so there are not many queries. One index gets bulk updated about every 2 hours.
>
> One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been giving us troubles, accepting connections but not processing them. Occasionaly dumps large 10GB+ heapdumps.
>
> After the last restart of that node (reallocation still enabled), two nodes attempt to move shards to it, but they stall part way. There has been no progress in the past day and the restarted node still contains no active shards.
>
> The gist provides the cluster stats, node stats, and the jstack of the three servers involved in the reallocation.
>
> [Reallocation failure · GitHub](https://gist.github.com/fb44ff1122fbf293ba5d)
>
> Cheers,
>
> Ivan
>
> --
--
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [October 2, 2012, 6:43pm UTC](https://discuss.elastic.co/t/shards-fail-to-reallocate/9212/4 "2012-10-02T18:43:31Z")
Is there a memory issue? I cannot get stats directly from ES for that
node, but the OS shows plenty of memory:
$ cat /proc/meminfo
> MemTotal: 24604156 kB
> MemFree: 5863904 kB
Running ES using the tanuki wrapper with 16gb allocated to the JVM:
-Xmx16384m
It appears that the JVM is using all of its allocated memory without using
the external memory.
top - 11:40:21 up 53 days, 21:40, 1 user, load average: 0.25, 0.26, 0.20
> Tasks: 128 total, 1 running, 127 sleeping, 0 stopped, 0 zombie
> Cpu(s): 4.0%us, 0.1%sy, 0.0%ni, 95.6%id, 0.3%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Mem: 24604156k total, 18740160k used, 5863996k free, 165644k buffers
> Swap: 17203192k total, 12940k used, 17190252k free, 847548k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4595 elastics 20 0 18.6g 16g 10m S 73.2 70.2 840:12.11 java
Or am I missing something? Is there anything of interest in the jstack
output?
Cheers,
Ivan
On Tue, Oct 2, 2012 at 11:25 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> I did not understand the misbehavior part, do you mean that shards fail to
> relocate to it? Or the fact that it has memory problems?
>
> On Oct 2, 2012, at 1:32 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> I should add that the index with the shard issue is not being queried
> against and not receiving any updates. It is merely an older version of the
> current index.
>
> The overall issue is not why these shards are not reallocating, but why is
> this node misbehaving?
>
> On Tue, Oct 2, 2012 at 10:19 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > I have a 12 node cluster running 0.19.8 with two-three 100gb indices that
> > have between six-eight shards and one replica. Not in production, so there
> > are not many queries. One index gets bulk updated about every 2 hours.
> >
> > One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been
> > giving us troubles, accepting connections but not processing them.
> > Occasionaly dumps large 10GB+ heapdumps.
> >
> > After the last restart of that node (reallocation still enabled), two
> > nodes attempt to move shards to it, but they stall part way. There has been
> > no progress in the past day and the restarted node still contains no active
> > shards.
> >
> > The gist provides the cluster stats, node stats, and the jstack of the
> > three servers involved in the reallocation.
> >
> > [Reallocation failure · GitHub](https://gist.github.com/fb44ff1122fbf293ba5d)
> >
> > Cheers,
> >
> > Ivan
>
> --
>
> --
--
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [October 2, 2012, 7:21pm UTC](https://discuss.elastic.co/t/shards-fail-to-reallocate/9212/5 "2012-10-02T19:21:28Z")
I just did some index cleanup. I first cleared the caches of the two older
indices. Cache size dropped almost 75%, but no change in the shards.
I then removed replicas for those two indices, and closed the oldest one.
Heap + cache size on all nodes dropped tremendously and all shards finally
relocated. The misbehaving node has 1 shard and is responding to requests.
--
Ivan
On Tue, Oct 2, 2012 at 10:32 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> I should add that the index with the shard issue is not being queried
> against and not receiving any updates. It is merely an older version of the
> current index.
>
> The overall issue is not why these shards are not reallocating, but why is
> this node misbehaving?
>
> On Tue, Oct 2, 2012 at 10:19 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > I have a 12 node cluster running 0.19.8 with two-three 100gb indices that
> > have between six-eight shards and one replica. Not in production, so there
> > are not many queries. One index gets bulk updated about every 2 hours.
> >
> > One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been
> > giving us troubles, accepting connections but not processing them.
> > Occasionaly dumps large 10GB+ heapdumps.
> >
> > After the last restart of that node (reallocation still enabled), two
> > nodes attempt to move shards to it, but they stall part way. There has been
> > no progress in the past day and the restarted node still contains no active
> > shards.
> >
> > The gist provides the cluster stats, node stats, and the jstack of the
> > three servers involved in the reallocation.
> >
> > [Reallocation failure · GitHub](https://gist.github.com/fb44ff1122fbf293ba5d)
> >
> > Cheers,
> >
> > Ivan
--
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [October 2, 2012, 9:36pm UTC](https://discuss.elastic.co/t/shards-fail-to-reallocate/9212/6 "2012-10-02T21:36:05Z")
Interesting update. The cluster is actually NOT using 0.19.8 but 0.19.2
instead. ES 0.19.8 was installed, but never used. The only node using
0.19.8 is the "misbehaving" one: srch-lv105.
I have encountered issues before with mismatched minor versions:
[https://groups.google.com/d/msg/elasticsearch/QL1fbGvLtnM/hW4j-H1LpfcJ](https://groups.google.com/d/msg/elasticsearch/QL1fbGvLtnM/hW4j-H1LpfcJ)
Going to downgrade the one node first before upgrading to 0.20.0 next week.
--
Ivan
On Tue, Oct 2, 2012 at 12:21 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> I just did some index cleanup. I first cleared the caches of the two older
> indices. Cache size dropped almost 75%, but no change in the shards.
>
> I then removed replicas for those two indices, and closed the oldest one.
> Heap + cache size on all nodes dropped tremendously and all shards finally
> relocated. The misbehaving node has 1 shard and is responding to requests.
>
> --
> Ivan
>
> On Tue, Oct 2, 2012 at 10:32 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > I should add that the index with the shard issue is not being queried
> > against and not receiving any updates. It is merely an older version of the
> > current index.
> >
> > The overall issue is not why these shards are not reallocating, but why
> > is this node misbehaving?
> >
> > On Tue, Oct 2, 2012 at 10:19 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> >
> > > I have a 12 node cluster running 0.19.8 with two-three 100gb indices
> > > that have between six-eight shards and one replica. Not in production, so
> > > there are not many queries. One index gets bulk updated about every 2 hours.
> > >
> > > One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been
> > > giving us troubles, accepting connections but not processing them.
> > > Occasionaly dumps large 10GB+ heapdumps.
> > >
> > > After the last restart of that node (reallocation still enabled), two
> > > nodes attempt to move shards to it, but they stall part way. There has been
> > > no progress in the past day and the restarted node still contains no active
> > > shards.
> > >
> > > The gist provides the cluster stats, node stats, and the jstack of the
> > > three servers involved in the reallocation.
> > >
> > > [Reallocation failure · GitHub](https://gist.github.com/fb44ff1122fbf293ba5d)
> > >
> > > Cheers,
> > >
> > > Ivan
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:10am UTC](https://discuss.elastic.co/t/shards-fail-to-reallocate/9212/7 "2017-07-06T03:10:28Z")