Shards fail to reallocate

Ivan · October 2, 2012, 5:19pm

I have a 12 node cluster running 0.19.8 with two-three 100gb indices that
have between six-eight shards and one replica. Not in production, so there
are not many queries. One index gets bulk updated about every 2 hours.

One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been giving
us troubles, accepting connections but not processing them. Occasionaly
dumps large 10GB+ heapdumps.

After the last restart of that node (reallocation still enabled), two nodes
attempt to move shards to it, but they stall part way. There has been no
progress in the past day and the restarted node still contains no active
shards.

The gist provides the cluster stats, node stats, and the jstack of the
three servers involved in the reallocation.

gist.github.com

https://gist.github.com/brusic/fb44ff1122fbf293ba5d

_nodesstats.json

{
  "cluster_name": "ESCluster",
  "nodes": {
    "BKkTlb2lQQKIWfJDW41afQ": {
      "name": "srch-lv113",
      "transport_address": "inet[/192.168.52.163:9300]",
      "indices": {
        "store": {
          "size": "114.7gb",
          "size_in_bytes": 123220409604

This file has been truncated. show original

_stats.json

{
  "ok": true,
  "_shards": {
    "total": 48,
    "successful": 48,
    "failed": 0
  },
  "_all": {
    "primaries": {
      "docs": {

This file has been truncated. show original

jstack 105

2012-10-02 10:02:12
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode):

"elasticsearch[srch-lv105][generic][T#3890]" daemon prio=10 tid=0x0000000041876000 nid=0x32d1 waiting on condition [0x00007fc6c1dcc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000400300f70> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)

This file has been truncated. show original

There are more than three files. show original

Cheers,

Ivan

--

Ivan · October 2, 2012, 5:32pm

I should add that the index with the shard issue is not being queried
against and not receiving any updates. It is merely an older version of the
current index.

The overall issue is not why these shards are not reallocating, but why is
this node misbehaving?

On Tue, Oct 2, 2012 at 10:19 AM, Ivan Brusic ivan@brusic.com wrote:

I have a 12 node cluster running 0.19.8 with two-three 100gb indices that
have between six-eight shards and one replica. Not in production, so there
are not many queries. One index gets bulk updated about every 2 hours.

One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been
giving us troubles, accepting connections but not processing them.
Occasionaly dumps large 10GB+ heapdumps.

After the last restart of that node (reallocation still enabled), two
nodes attempt to move shards to it, but they stall part way. There has been
no progress in the past day and the restarted node still contains no active
shards.

The gist provides the cluster stats, node stats, and the jstack of the
three servers involved in the reallocation.

Reallocation failure · GitHub

Cheers,

Ivan

--

kimchy · October 2, 2012, 6:25pm

I did not understand the misbehavior part, do you mean that shards fail to relocate to it? Or the fact that it has memory problems?

On Oct 2, 2012, at 1:32 PM, Ivan Brusic ivan@brusic.com wrote:

I should add that the index with the shard issue is not being queried against and not receiving any updates. It is merely an older version of the current index.

The overall issue is not why these shards are not reallocating, but why is this node misbehaving?

On Tue, Oct 2, 2012 at 10:19 AM, Ivan Brusic ivan@brusic.com wrote:
I have a 12 node cluster running 0.19.8 with two-three 100gb indices that have between six-eight shards and one replica. Not in production, so there are not many queries. One index gets bulk updated about every 2 hours.

One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been giving us troubles, accepting connections but not processing them. Occasionaly dumps large 10GB+ heapdumps.

After the last restart of that node (reallocation still enabled), two nodes attempt to move shards to it, but they stall part way. There has been no progress in the past day and the restarted node still contains no active shards.

The gist provides the cluster stats, node stats, and the jstack of the three servers involved in the reallocation.

Reallocation failure · GitHub

Cheers,

Ivan

--

--

Ivan · October 2, 2012, 6:43pm

Is there a memory issue? I cannot get stats directly from ES for that
node, but the OS shows plenty of memory:

$ cat /proc/meminfo

MemTotal: 24604156 kB
MemFree: 5863904 kB

Running ES using the tanuki wrapper with 16gb allocated to the JVM:
-Xmx16384m

It appears that the JVM is using all of its allocated memory without using
the external memory.

top - 11:40:21 up 53 days, 21:40, 1 user, load average: 0.25, 0.26, 0.20

Tasks: 128 total, 1 running, 127 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.0%us, 0.1%sy, 0.0%ni, 95.6%id, 0.3%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 24604156k total, 18740160k used, 5863996k free, 165644k buffers
Swap: 17203192k total, 12940k used, 17190252k free, 847548k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4595 elastics 20 0 18.6g 16g 10m S 73.2 70.2 840:12.11 java

Or am I missing something? Is there anything of interest in the jstack
output?

Cheers,

Ivan

On Tue, Oct 2, 2012 at 11:25 AM, Shay Banon kimchy@gmail.com wrote:

I did not understand the misbehavior part, do you mean that shards fail to
relocate to it? Or the fact that it has memory problems?

On Oct 2, 2012, at 1:32 PM, Ivan Brusic ivan@brusic.com wrote:

I should add that the index with the shard issue is not being queried
against and not receiving any updates. It is merely an older version of the
current index.

The overall issue is not why these shards are not reallocating, but why is
this node misbehaving?

On Tue, Oct 2, 2012 at 10:19 AM, Ivan Brusic ivan@brusic.com wrote:

I have a 12 node cluster running 0.19.8 with two-three 100gb indices that
have between six-eight shards and one replica. Not in production, so there
are not many queries. One index gets bulk updated about every 2 hours.

One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been
giving us troubles, accepting connections but not processing them.
Occasionaly dumps large 10GB+ heapdumps.

After the last restart of that node (reallocation still enabled), two
nodes attempt to move shards to it, but they stall part way. There has been
no progress in the past day and the restarted node still contains no active
shards.

The gist provides the cluster stats, node stats, and the jstack of the
three servers involved in the reallocation.

Reallocation failure · GitHub

Cheers,

Ivan

--

--

--

Ivan · October 2, 2012, 7:21pm

I just did some index cleanup. I first cleared the caches of the two older
indices. Cache size dropped almost 75%, but no change in the shards.

I then removed replicas for those two indices, and closed the oldest one.
Heap + cache size on all nodes dropped tremendously and all shards finally
relocated. The misbehaving node has 1 shard and is responding to requests.

--
Ivan

On Tue, Oct 2, 2012 at 10:32 AM, Ivan Brusic ivan@brusic.com wrote:

I should add that the index with the shard issue is not being queried
against and not receiving any updates. It is merely an older version of the
current index.

The overall issue is not why these shards are not reallocating, but why is
this node misbehaving?

On Tue, Oct 2, 2012 at 10:19 AM, Ivan Brusic ivan@brusic.com wrote:

I have a 12 node cluster running 0.19.8 with two-three 100gb indices that
have between six-eight shards and one replica. Not in production, so there
are not many queries. One index gets bulk updated about every 2 hours.

One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been
giving us troubles, accepting connections but not processing them.
Occasionaly dumps large 10GB+ heapdumps.

After the last restart of that node (reallocation still enabled), two
nodes attempt to move shards to it, but they stall part way. There has been
no progress in the past day and the restarted node still contains no active
shards.

The gist provides the cluster stats, node stats, and the jstack of the
three servers involved in the reallocation.

Reallocation failure · GitHub

Cheers,

Ivan

--

Ivan · October 2, 2012, 9:36pm

Interesting update. The cluster is actually NOT using 0.19.8 but 0.19.2
instead. ES 0.19.8 was installed, but never used. The only node using
0.19.8 is the "misbehaving" one: srch-lv105.

I have encountered issues before with mismatched minor versions:
https://groups.google.com/d/msg/elasticsearch/QL1fbGvLtnM/hW4j-H1LpfcJ

Going to downgrade the one node first before upgrading to 0.20.0 next week.

--
Ivan

On Tue, Oct 2, 2012 at 12:21 PM, Ivan Brusic ivan@brusic.com wrote:

I just did some index cleanup. I first cleared the caches of the two older
indices. Cache size dropped almost 75%, but no change in the shards.

I then removed replicas for those two indices, and closed the oldest one.
Heap + cache size on all nodes dropped tremendously and all shards finally
relocated. The misbehaving node has 1 shard and is responding to requests.

--
Ivan

On Tue, Oct 2, 2012 at 10:32 AM, Ivan Brusic ivan@brusic.com wrote:

I should add that the index with the shard issue is not being queried
against and not receiving any updates. It is merely an older version of the
current index.

The overall issue is not why these shards are not reallocating, but why
is this node misbehaving?

On Tue, Oct 2, 2012 at 10:19 AM, Ivan Brusic ivan@brusic.com wrote:

I have a 12 node cluster running 0.19.8 with two-three 100gb indices
that have between six-eight shards and one replica. Not in production, so
there are not many queries. One index gets bulk updated about every 2 hours.

One node in particular (srch-lv105, X30RJ0i-QFOfNrvHT291tw) has been
giving us troubles, accepting connections but not processing them.
Occasionaly dumps large 10GB+ heapdumps.

After the last restart of that node (reallocation still enabled), two
nodes attempt to move shards to it, but they stall part way. There has been
no progress in the past day and the restarted node still contains no active
shards.

The gist provides the cluster stats, node stats, and the jstack of the
three servers involved in the reallocation.

Reallocation failure · GitHub

Cheers,

Ivan

--