Elasticsearch version: 5.0.1
Plugins installed: [ik]
JVM version: 1.8.0_111
OS version: Ubuntu 16.04.1 LTS
Description of the problem including expected versus actual behavior:
Just restart primary node in cluster, sometimes it cost more than 10min to start.
The older version of es (like 1.7), do not need so much time. Is there any way to solve?
After startup, it seems GC frequently, with logs below. But about 10 min later, it works fine.
[2016-12-12T14:23:09,078][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1519] overhead, spent [315ms] collecting in the last [1s]
[2016-12-12T14:23:10,079][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1520] overhead, spent [350ms] collecting in the last [1s]
[2016-12-12T14:23:11,079][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1521] overhead, spent [318ms] collecting in the last [1s]
[2016-12-12T14:23:12,080][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1522] overhead, spent [276ms] collecting in the last [1s]
[2016-12-12T14:23:13,085][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1523] overhead, spent [318ms] collecting in the last [1s]
[2016-12-12T14:23:14,126][INFO ][o.e.m.j.JvmGcMonitorService] [FaFejOC] [gc][1524] overhead, spent [269ms] collecting in the last [1s]
Provide logs (if relevant):
output after type: systemctl restart elasticsearch.service, you can find it initializing 10 min later..
[2016-12-12T13:44:43,947][INFO ][o.e.n.Node ] [FaFejOC] stopped
[2016-12-12T13:44:43,948][INFO ][o.e.n.Node ] [FaFejOC] closing ...
[2016-12-12T13:44:44,062][INFO ][o.e.n.Node ] [FaFejOC] closed
[2016-12-12T13:57:10,647][INFO ][o.e.n.Node ] [] initializing ...
[2016-12-12T13:57:10,821][INFO ][o.e.e.NodeEnvironment ] [FaFejOC] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [11gb], net total_space [19.5gb], spins? [possibly], types [ext4]
[2016-12-12T13:57:10,822][INFO ][o.e.e.NodeEnvironment ] [FaFejOC] heap size [15.9gb], compressed ordinary object pointers [true]
[2016-12-12T13:57:11,036][INFO ][o.e.n.Node ] [FaFejOC] node name [FaFejOC] derived from node ID; set [node.name] to override
[2016-12-12T13:57:11,040][INFO ][o.e.n.Node ] [FaFejOC] version[5.0.1], pid[11582], build[080bb47/2016-11-11T22:08:49.812Z], OS[Linux/4.4.0-34-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_111/25.111-b14]
elasticsearch.yml conf:
discovery.zen.ping.unicast.hosts: ["prd-es-1", "prd-es-2", "prd-es-3"]
discovery.zen.minimum_master_nodes: 2
node.max_local_storage_nodes: 1
bootstrap.memory_lock: true
GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-Xms16g
-Xmx16g