# 3-node cluster dying intermittently with CPU spike **URL:** \ **Category:** Elasticsearch\ **Created:** [January 5, 2017, 12:05am UTC](https://discuss.elastic.co/t/3-node-cluster-dying-intermittently-with-cpu-spike/70621 "2017-01-05T00:05:55Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![nwood888](https://avatars.discourse-cdn.com/v4/letter/n/71c47a/32.png) [@nwood888](https://discuss.elastic.co/u/nwood888)\ **Post date:** [January 5, 2017, 12:05am UTC](https://discuss.elastic.co/t/3-node-cluster-dying-intermittently-with-cpu-spike/70621/1 "2017-01-05T00:05:55Z")
I have a 3-node 5.1.1 cluster that I believe is sized correctly. In the past 2 weeks, it's locked up twice, ultimately reaching the queue capacity of 1000 and requiring a full cluster restart to bring it back online. Elastic is running inside docker containers on 3 ec2 instances in different availability zones. I believe these are the only relevant Elastic logs from when this happened: > [2017-01-04T13:40:44,434][INFO][o.e.m.j.JvmGcMonitorService] [es1] [gc][1118938] overhead, spent [294ms] collecting in the last [1.1s] > [2017-01-04T13:42:29,514][INFO][o.e.c.m.MetaDataMappingService] [es1] [.monitoring-kibana-2-2017.01.04/0kTB19GeRg686SSUtC8l5g] update\_mapping [kibana\_stats] > [2017-01-04T13:43:33,268][WARN][o.e.m.j.JvmGcMonitorService] [es1] [gc][1119105] overhead, spent [1.9s] collecting in the last [2.7s] > [2017-01-04T13:43:46,771][INFO][o.e.n.Node] [es1] initializing ... > [2017-01-04T13:43:46,841][INFO][o.e.e.NodeEnvironment] [es1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvdf1)]], net usable\_space [134.9gb], net total\_space [147.5gb], spins? [possibly], types [ext4] > [2017-01-04T13:43:46,842][INFO][o.e.e.NodeEnvironment] [es1] heap size [6.9gb], compressed ordinary object pointers [true] > [2017-01-04T13:43:46,911][INFO][o.e.n.Node] [es1] node name [es1], node ID [3nX\_\_vT2Q4aA-XjLh9m7Lw] > [2017-01-04T13:43:46,913][INFO][o.e.n.Node] [es1] version[5.1.1], pid[10], build[5395e21/2016-12-06T12:36:15.409Z], OS[Linux/4.4.35-33.55.amzn1.x86\_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0\_111/25.111-b14] My Zabbix graphs show a CPU spike starting right around 13:40 on all 3 nodes. It looks like that might correspond with garbage collection from the logs, but I'm not sure. It then appears that the docker container restarted on it's own. I haven't found anything in the system logs to indicate why that might have happened. Looking at the marvel graphs for the first (master) node at 13:40, heap was ~5GB out of 7GB. CPU was bumping around between 0 and 25% for the previous several hours. Index and search latency were low. Segment count was ~600. Some thoughts that I've had but haven't finished exploring: - Maybe this is caused by a rogue query? If so, any tips on tracking this down? - The docker MTU is 1500 and the host primary network interface MTU is 9001. I've seen older references to this causing connectivity problems between nodes, but it seems like those issues might be worked out in recent version of Elastic and Docker. I'm wondering how to troubleshoot this further as I'm not seeing anything extra in the logs. Any direction would be much appreciated. Nick ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2017, 12:10am UTC](https://discuss.elastic.co/t/3-node-cluster-dying-intermittently-with-cpu-spike/70621/2 "2017-01-05T00:10:29Z")
How many indices, shards and how much data? What JVM? ---
**Author:** ![nwood888](https://avatars.discourse-cdn.com/v4/letter/n/71c47a/32.png) [@nwood888](https://discuss.elastic.co/u/nwood888)\ **Post date:** [January 5, 2017, 12:36am UTC](https://discuss.elastic.co/t/3-node-cluster-dying-intermittently-with-cpu-spike/70621/3 "2017-01-05T00:36:38Z")
44 indices 289 shards 23.3M documents 16GB data These are c4.2xlarge instance types Using the official "elasticsearch:5" docker image JVM is: Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0\_111/25.111-b14 ---
**Author:** ![tronicum](https://avatars.discourse-cdn.com/v4/letter/t/5f8ce5/32.png) [@tronicum](https://discuss.elastic.co/u/tronicum)\ **Post date:** [January 12, 2017, 9:44am UTC](https://discuss.elastic.co/t/3-node-cluster-dying-intermittently-with-cpu-spike/70621/4 "2017-01-12T09:44:23Z")
I got this if i stream too much data into a docker cluster (1.8 gb access logs on a vmware with 24gb ram and 4 cpus) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 9, 2017, 9:44am UTC](https://discuss.elastic.co/t/3-node-cluster-dying-intermittently-with-cpu-spike/70621/5 "2017-02-09T09:44:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.