Elasticsearch hangs

Elastic Stack Elasticsearch
1

hi,all,

  • *I have 4 es node with 32 shard 1 replica, each with 8G memory. my
    elasticsearch version is elasticsearch-0.18.5.
    I start each node with command:
    nohup bin/elasticsearch -Xms8g -Xmx8g -Des.max-open-files=true
    -Dbootstrap.mlockall=true -p es.pid

and current the index size is:
size: 35.3gb (70.7gb)
docs: 15938184 (17506608)

from elasticsearch-head i noticed that:
jvm: {

  • pid: 17608
  • version: 1.6.0_13
  • vm_name: Java HotSpot(TM) 64-Bit Server VM
  • vm_version: 11.3-b02
  • vm_vendor: Sun Microsystems Inc.
  • start_time: 1337151263158
  • mem: {
    • heap_init: 2gb
    • heap_init_in_bytes: 2147483648
    • heap_max: 5.3gb
    • heap_max_in_bytes: 5726666752
    • non_heap_init: 130.4mb
    • non_heap_init_in_bytes: 136773632
    • non_heap_max: 560mb
    • non_heap_max_in_bytes: 587202560
      }

}

from top command:
21003 work 18 0 8879m 8.3g 9796 S 257.0 13.2 48:45.98 java

21003 is elasticsearch's pid and we can see that cpu usage is 257.0% and
keeps around that, some time even 300+%.

every one month I have to restarted all my es nodes otherwise it will
responses very slow and dumps a *.hprof file.

from the index size we can see that we have only 30G around index(70 total
with replica) and each node has 8G memeory, but es still performs very
poorly.

how should i do to achieve better performance and avoid restarting es
nodes(which is very frustrating)

2

It might be worthwhile to upgrade to the latest version 0.19.7. We had a very similar problem with 0.19.3 & 0.19.4 and upgrading to 0.19.7 fixed this issue for us. Take a look at the upgrade notes, concerning upgrading from 0.18.X versions.

Regards,

Daniel.

Am 28.06.2012 um 11:37 schrieb Weiwei Wang:

hi,all,
I have 4 es node with 32 shard 1 replica, each with 8G memory. my elasticsearch version is elasticsearch-0.18.5.
I start each node with command:
nohup bin/elasticsearch -Xms8g -Xmx8g -Des.max-open-files=true -Dbootstrap.mlockall=true -p es.pid

and current the index size is:
size: 35.3gb (70.7gb)
docs: 15938184 (17506608)

from elasticsearch-head i noticed that:
jvm: {
pid: 17608
version: 1.6.0_13
vm_name: Java HotSpot(TM) 64-Bit Server VM
vm_version: 11.3-b02
vm_vendor: Sun Microsystems Inc.
start_time: 1337151263158
mem: {
heap_init: 2gb
heap_init_in_bytes: 2147483648
heap_max: 5.3gb
heap_max_in_bytes: 5726666752
non_heap_init: 130.4mb
non_heap_init_in_bytes: 136773632
non_heap_max: 560mb
non_heap_max_in_bytes: 587202560
}
}

from top command:
21003 work 18 0 8879m 8.3g 9796 S 257.0 13.2 48:45.98 java

21003 is elasticsearch's pid and we can see that cpu usage is 257.0% and keeps around that, some time even 300+%.

every one month I have to restarted all my es nodes otherwise it will responses very slow and dumps a *.hprof file.

from the index size we can see that we have only 30G around index(70 total with replica) and each node has 8G memeory, but es still performs very poorly.

how should i do to achieve better performance and avoid restarting es nodes(which is very frustrating)

3

thanks daniel, i'm planning to do that. I'm testing my client which use
client lib's TransportClient to connect to es cluster and find that client
lib also need to upgrade, so I need to do a full test because we use it in
our online product.

On Thursday, June 28, 2012 7:20:53 PM UTC+8, Daniel Schnell wrote:

It might be worthwhile to upgrade to the latest version 0.19.7. We had a
very similar problem with 0.19.3 & 0.19.4 and upgrading to 0.19.7 fixed
this issue for us. Take a look at the upgrade notes, concerning upgrading
from 0.18.X versions.

Regards,

Daniel.

Am 28.06.2012 um 11:37 schrieb Weiwei Wang:

hi,all,

  • *I have 4 es node with 32 shard 1 replica, each with 8G memory. my
    elasticsearch version is elasticsearch-0.18.5.
    I start each node with command:
    nohup bin/elasticsearch -Xms8g -Xmx8g -Des.max-open-files=true
    -Dbootstrap.mlockall=true -p es.pid

and current the index size is:
size: 35.3gb (70.7gb)
docs: 15938184 (17506608)

from elasticsearch-head i noticed that:
jvm: {

  • pid: 17608
  • version: 1.6.0_13
  • vm_name: Java HotSpot(TM) 64-Bit Server VM
  • vm_version: 11.3-b02
  • vm_vendor: Sun Microsystems Inc.
  • start_time: 1337151263158
  • mem: {
    • heap_init: 2gb
    • heap_init_in_bytes: 2147483648
    • heap_max: 5.3gb
    • heap_max_in_bytes: 5726666752
    • non_heap_init: 130.4mb
    • non_heap_init_in_bytes: 136773632
    • non_heap_max: 560mb
    • non_heap_max_in_bytes: 587202560
      }

}

from top command:
21003 work 18 0 8879m 8.3g 9796 S 257.0 13.2 48:45.98 java

21003 is elasticsearch's pid and we can see that cpu usage is 257.0% and
keeps around that, some time even 300+%.

every one month I have to restarted all my es nodes otherwise it will
responses very slow and dumps a *.hprof file.

from the index size we can see that we have only 30G around index(70 total
with replica) and each node has 8G memeory, but es still performs very
poorly.

how should i do to achieve better performance and avoid restarting es
nodes(which is very frustrating)