# Kibana and Elasticsearch Keep crashing **URL:** \ **Category:** Kibana\ **Created:** [October 12, 2016, 3:21pm UTC](https://discuss.elastic.co/t/kibana-and-elasticsearch-keep-crashing/62826 "2016-10-12T15:21:33Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![upendra](https://avatars.discourse-cdn.com/v4/letter/u/e68b1a/32.png) [@upendra](https://discuss.elastic.co/u/upendra)\ **Post date:** [October 12, 2016, 3:21pm UTC](https://discuss.elastic.co/t/kibana-and-elasticsearch-keep-crashing/62826/1 "2016-10-12T15:21:33Z")
HI, Our elastic search cluster and Kibana keep crashing when we execute reports. The following are the product versions: 1. Logstash: 2.4.0 2. Elasticsearch: 2.4.0 3. Kibana: 4.6.1 4. Java: 1.8.0 The following is the error that we get: ![](https://us1.discourse-cdn.com/elastic/original/2X/5/578ca1e71cfed539eb176bfa4cfb2888202ee9aa.png) Our cluster design is as follows: 1. Logstash Inputs: 4 2. Logstash output: 1 3. ES Master & Data: 5 ( Each one is both master and Data) 4. ES Client node (with Kibana): 1 ELK cluster is on Centos 7 each with 16 GB RAM. Out of which 4GB is allotted ES\_HEAP\_SIZE parameter. We have also tried setting the Node Option parameter to: exec "{NODE}" --max-old-space-size=100 "{DIR}/src/cli" ${@} But still our Elasticsearch and Kibana keep crashing. Thanks, Upendra ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 14, 2016, 10:03am UTC](https://discuss.elastic.co/t/kibana-and-elasticsearch-keep-crashing/62826/2 "2016-10-14T10:03:40Z")
What's in your logs? ---
**Author:** ![upendra](https://avatars.discourse-cdn.com/v4/letter/u/e68b1a/32.png) [@upendra](https://discuss.elastic.co/u/upendra)\ **Post date:** [October 14, 2016, 10:55am UTC](https://discuss.elastic.co/t/kibana-and-elasticsearch-keep-crashing/62826/3 "2016-10-14T10:55:29Z")
Hi Mark, Running command jmap -heap pid gives the following: Attaching to process ID 15144, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.101-b13 using parallel threads in the new generation. using thread-local object allocation. Concurrent Mark-Sweep GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 8589934592 (8192.0MB) NewSize = 348913664 (332.75MB) MaxNewSize = 348913664 (332.75MB) OldSize = 8241020928 (7859.25MB) NewRatio = 2 SurvivorRatio = 8 MetaspaceSize = 21807104 (20.796875MB) CompressedClassSpaceSize = 1073741824 (1024.0MB) MaxMetaspaceSize = 17592186044415 MB G1HeapRegionSize = 0 (0.0MB) Heap Usage: New Generation (Eden + 1 Survivor Space): capacity = 314048512 (299.5MB) used = 314048496 (299.49998474121094MB) free = 16 (1.52587890625E-5MB) 99.99999490524573% used Eden Space: capacity = 279183360 (266.25MB) used = 279183360 (266.25MB) free = 0 (0.0MB) 100.0% used From Space: capacity = 34865152 (33.25MB) used = 34865136 (33.24998474121094MB) free = 16 (1.52587890625E-5MB) 99.99995410890507% used To Space: capacity = 34865152 (33.25MB) used = 0 (0.0MB) free = 34865152 (33.25MB) 0.0% used concurrent mark-sweep generation: capacity = 8241020928 (7859.25MB) used = 8241020896 (7859.249969482422MB) free = 32 (3.0517578125E-5MB) 99.9999996116986% used 15745 interned Strings occupying 2446000 bytes. Thanks, Upendra Thanks, Upendra ---
**Author:** ![upendra](https://avatars.discourse-cdn.com/v4/letter/u/e68b1a/32.png) [@upendra](https://discuss.elastic.co/u/upendra)\ **Post date:** [October 14, 2016, 11:01am UTC](https://discuss.elastic.co/t/kibana-and-elasticsearch-keep-crashing/62826/4 "2016-10-14T11:01:37Z")
Hi Mark, Please see the logs. i am unable to send you complete logs due to space constraints of this forum. ![](https://us1.discourse-cdn.com/elastic/original/2X/b/b007448b1f10cdc0f26895a34dc86091e7a985ea.png) Thanks, Upendra ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [October 14, 2016, 11:21am UTC](https://discuss.elastic.co/t/kibana-and-elasticsearch-keep-crashing/62826/5 "2016-10-14T11:21:15Z")
Hey, please use [gist](https://gist.github.com) or other pastebins to put some logs somewhere (also make sure they dont contain sensitive information) - and keep the format as text. Thanks! --Alex ---
**Author:** ![upendra](https://avatars.discourse-cdn.com/v4/letter/u/e68b1a/32.png) [@upendra](https://discuss.elastic.co/u/upendra)\ **Post date:** [October 14, 2016, 12:11pm UTC](https://discuss.elastic.co/t/kibana-and-elasticsearch-keep-crashing/62826/6 "2016-10-14T12:11:54Z")
Thanks Alex for that help. Please find the log entry here : > Regards, Upendra ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [October 14, 2016, 12:29pm UTC](https://discuss.elastic.co/t/kibana-and-elasticsearch-keep-crashing/62826/7 "2016-10-14T12:29:27Z")
If you read that log, you can spot an out of memory exception. This means you have to restart your node immediately, as the behaviour after such an exception is not specified (you just dont know if everything works or not). However in order to prevent those issues in the future, you should find out what triggers this exception. Is it a special query? You might want to read the following docs regarding to that topic > **[Limiting Memory Usage | Elasticsearch: The Definitive Guide \[2.x\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html)** > **[Heap: Sizing and Swapping | Elasticsearch: The Definitive Guide \[2.x\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html)** > **[Six Ways to Crash Elasticsearch](https://www.elastic.co/blog/found-crash-elasticsearch)** > > As much as we love Elasticsearch, at Found we've seen customers crash their clusters in numerous ways. Mostly due to simple misunderstandings and usually the fixes are fairly straightforward. In our quest to enlighten new adopters and entertain the... > [@Getting OOME's (Out of Memory Exceptions) to stop](https://discuss.elastic.co/t/getting-oomes-out-of-memory-exceptions-to-stop/18332): > > I have a 10 machine cluster where frequently (about once per day when indexing and querying is at its height) one elasticsearch node goes OOM... It usually recovers, but by this time the cluster is redistributing the lost shards, which causes more load, which often in turn causes an OOM on another machine. Each machine has 32GB memory of which I currently have 12GB allocated to Elasticsearch. I have logstash (max 500M) and redis (max 2GB) running on the machines too, and see that the rem… You can use the cat APIs or monitoring to see if you have continously rising memory usages or spikes which cause this behaviour. Hope this helps. --Alex ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 15, 2016, 2:30am UTC](https://discuss.elastic.co/t/kibana-and-elasticsearch-keep-crashing/62826/8 "2016-10-15T02:30:49Z")
You should definitely be using Marvel as well. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:40pm UTC](https://discuss.elastic.co/t/kibana-and-elasticsearch-keep-crashing/62826/9 "2017-07-06T13:40:18Z")