Elasticsearch version:5.2.2
JVM version:1.8.0_111
Os version:centos7.2
I deploy 5 client nodes to transfer data,but when i update version to 5.2.2,i find the client node do full gc frequently.
[2017-03-24T11:38:07,458][WARN ][o.e.m.j.JvmGcMonitorService] [es154.162.client2] [gc][816] overhead, spent [925ms] collecting in the last [1s]
[2017-03-24T11:38:12,327][WARN ][o.e.m.j.JvmGcMonitorService] [es154.162.client2] [gc][820] overhead, spent [956ms] collecting in the last [1.8s]
[2017-03-24T11:38:14,482][WARN ][o.e.m.j.JvmGcMonitorService] [es154.162.client2] [gc][822] overhead, spent [948ms] collecting in the last [1.1s]
[2017-03-24T11:38:16,924][WARN ][o.e.m.j.JvmGcMonitorService] [es154.162.client2] [gc][823] overhead, spent [1.6s] collecting in the last [2.4s]
[2017-03-24T11:38:19,326][WARN ][o.e.m.j.JvmGcMonitorService] [es154.162.client2] [gc][824] overhead, spent [2.2s] collecting in the last [2.4s]
[2017-03-24T11:38:28,141][WARN ][o.e.m.j.JvmGcMonitorService] [es154.162.client2] [gc][825] overhead, spent [1s] collecting in the last [1.1s]
[2017-03-24T11:38:29,523][INFO ][o.e.m.j.JvmGcMonitorService] [es154.162.client2] [gc][826] overhead, spent [3.5s] collecting in the last [9s]
Here are the gc log:
95.285: [Full GC (Allocation Failure) 9865206K->9014457K(10289856K), 1.7207554 secs]
97.775: [Full GC (Allocation Failure) 10275503K->8547170K(10289856K), 1.4681195 secs]
99.255: [GC (CMS Initial Mark) 8563628K(10289856K), 0.0027359 secs]
99.433: [GC (CMS Final Remark) 8643527K(10289856K), 0.0345294 secs]
100.573: [Full GC (Allocation Failure) 10128580K->8570911K(10289856K), 1.6069245 secs]
102.195: [GC (CMS Initial Mark) 8587295K(10289856K), 0.0021058 secs]
102.322: [GC (CMS Final Remark) 9797549K(10289856K), 0.0304902 secs]
102.412: [Full GC (Allocation Failure) 10159238K->10115353K(10289856K), 1.8626932 secs]
104.317: [Full GC (Allocation Failure) 10273358K->10263489K(10289856K), 0.8585131 secs]
105.202: [Full GC (Allocation Failure) 10263489K->10258863K(10289856K), 2.5438956 secs]
114.049: [GC (CMS Initial Mark) 10265142K(10289856K), 0.0105399 secs]
114.093: [Full GC (Allocation Failure) 10272675K->10148248K(10289856K), 2.3793751 secs]
116.528: [Full GC (Allocation Failure) 10287412K->9211888K(10289856K), 2.6253769 secs]
119.180: [GC (CMS Initial Mark) 9238939K(10289856K), 0.0060089 secs]
Here are the setting:
node.attr.host: es162
node.attr.box_type: hot
cluster.routing.allocation.awareness.attributes: host
node.master: false
node.data: false
node.name: es162.client
transport.tcp.compress: true
network.host: [_local_,_site_]
discovery.zen.ping.unicast.hosts: ["xxx"]
indices.queries.cache.size: 50%
indices.fielddata.cache.size: 20%
thread_pool.bulk.queue_size: 1000