Restart primary node in cluster, startup very slow


(zhulin) #1

Elasticsearch version: 5.0.1

Plugins installed: [ik]

JVM version: 1.8.0_111

OS version: Ubuntu 16.04.1 LTS

Description of the problem including expected versus actual behavior:
Just restart primary node in cluster, sometimes it cost more than 10min to start.
The older version of es (like 1.7), do not need so much time. Is there any way to solve?

After startup, it seems GC frequently, with logs below. But about 10 min later, it works fine.

[2016-12-12T14:23:09,078][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1519] overhead, spent [315ms] collecting in the last [1s]
[2016-12-12T14:23:10,079][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1520] overhead, spent [350ms] collecting in the last [1s]
[2016-12-12T14:23:11,079][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1521] overhead, spent [318ms] collecting in the last [1s]
[2016-12-12T14:23:12,080][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1522] overhead, spent [276ms] collecting in the last [1s]
[2016-12-12T14:23:13,085][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1523] overhead, spent [318ms] collecting in the last [1s]
[2016-12-12T14:23:14,126][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1524] overhead, spent [269ms] collecting in the last [1s]
Provide logs (if relevant):

output after type: systemctl restart elasticsearch.service, you can find it initializing 10 min later..

[2016-12-12T13:44:43,947][INFO ][o.e.n.Node ] [FaFejOC] stopped
[2016-12-12T13:44:43,948][INFO ][o.e.n.Node ] [FaFejOC] closing ...
[2016-12-12T13:44:44,062][INFO ][o.e.n.Node ] [FaFejOC] closed

[2016-12-12T13:57:10,647][INFO ][o.e.n.Node ] [] initializing ...
[2016-12-12T13:57:10,821][INFO ][o.e.e.NodeEnvironment ] [FaFejOC] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [11gb], net total_space [19.5gb], spins? [possibly], types [ext4]
[2016-12-12T13:57:10,822][INFO ][o.e.e.NodeEnvironment ] [FaFejOC] heap size [15.9gb], compressed ordinary object pointers [true]
[2016-12-12T13:57:11,036][INFO ][o.e.n.Node ] [FaFejOC] node name [FaFejOC] derived from node ID; set [node.name] to override
[2016-12-12T13:57:11,040][INFO ][o.e.n.Node ] [FaFejOC] version[5.0.1], pid[11582], build[080bb47/2016-11-11T22:08:49.812Z], OS[Linux/4.4.0-34-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_111/25.111-b14]

elasticsearch.yml conf:

discovery.zen.ping.unicast.hosts: ["prd-es-1", "prd-es-2", "prd-es-3"]
discovery.zen.minimum_master_nodes: 2
node.max_local_storage_nodes: 1
bootstrap.memory_lock: true

GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

-Xms16g
-Xmx16g


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.