JVM pause,Elasticsearch stop

Elastic Stack Elasticsearch
1

hello:
We use Elasticsearch 2.3.2 and run one data nodes in server with 32G RAM and 32 Core CPU. Elasticsearch java heap size is set to 24G and the index is configured to 5 shards with 0 replica.
Bulk indexing around twenty-five thousand documents per seconds,Elasticsearch will stop after the index for some time.

See the GC info:
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT

100.00 100.00 100.00 64.45 98.31 96.99 585 68.597 6 0.434 69.031
100.00 100.00 100.00 64.45 98.31 96.99 585 68.597 6 0.434 69.031
100.00 100.00 100.00 64.45 98.31 96.99 585 68.597 6 0.434 69.031
100.00 100.00 100.00 64.45 98.31 96.99 585 68.597 6 0.434 69.031
100.00 100.00 100.00 64.45 98.31 96.99 585 68.597 6 0.434 69.031
100.00 100.00 100.00 64.45 98.31 96.99 585 68.597 6 0.434 69.031
100.00 100.00 100.00 64.45 98.31 96.99 585 68.597 6 0.434 69.031
100.00 100.00 100.00 64.45 98.31 96.99 585 68.597 6 0.434 69.031
100.00 100.00 100.00 64.45 98.31 96.99 585 68.597 6 0.434 69.031

Elasticsearch GC log show:

  • age 1: 131167760 bytes, 131167760 total
    : 1761742K->195904K(1763584K), 0.2424211 secs] 16479515K->14975389K(24969920K), 0.2426231 secs] [Times: user=5.04 sys=0.02, real=0.24 secs]
    2016-05-13T00:10:01.420+0800: 682.200: Total time for which application threads were stopped: 0.2442729 seconds, Stopping threads took: 0.0006558 seconds
    2016-05-13T00:10:02.221+0800: 683.001: [GC (Allocation Failure) 683.001: [ParNew
    Desired survivor size 100302848 bytes, new threshold 1 (max 6)
  • age 1: 148728784 bytes, 148728784 total
    : 1763584K->195904K(1763584K), 0.3199880 secs] 16543069K->15041803K(24969920K), 0.3608433 secs] [Times: user=1.80 sys=0.01, real=0.36 secs]
    2016-05-13T00:10:02.582+0800: 683.362: Total time for which application threads were stopped: 0.3627899 seconds, Stopping threads took: 0.0004477 seconds
    2016-05-13T00:10:02.584+0800: 683.364: Total time for which application threads were stopped: 0.0015803 seconds, Stopping threads took: 0.0002223 seconds
    2016-05-13T00:10:03.446+0800: 684.226: [GC (Allocation Failure) 684.472: [ParNew

Log for a long time to stop here,At this time ,i can not visit 'http://127.0.0.1:9200'.
After a long period of time after the log display：
[WARN ][monitor.jvm ] [gc][young][1253][1431] duration [28.9m], collections [1]/[29.1m], total [28.9m]/[31.5m], memory [15.7gb]->[14.6gb]/[23.8gb], all_pools {[young] [1.3gb]->[2.3mb]/[1.4gb]}{[survivor] [174.4mb]->[191.3mb]/[191.3mb]}{[old] [14.3gb]->[14.4gb]/[22.1gb]}

Young GC take 28 minutes,why?

I'm sorry my English is poor. Thanks!

2

I'd suggest that your node is overloaded and you need to add more heap to it, or add another node.

3

Thanks,after i will try to allocate more memory.

in this node,Eden space is not enough?Old Space only use 64.45%.
but in another node with 32G RAM and 24 Core CPU. Elasticsearch java heap size is set to 24G.
Elasticsearch configuration is the same,this node can not apper this question;but the disk read and write speed is slower than that,Will it be beacause of this?

Thanks