# OutOfMemoryError on adequately sized cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [May 9, 2018, 3:20pm UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192 "2018-05-09T15:20:26Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [May 9, 2018, 3:20pm UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/1 "2018-05-09T15:20:26Z")
Hi, running 6.2.4 Cluster topology: - 3 masters - 2 ingest nodes - 1 Coordinator - 3 Data nodes The masters and the ingest nodes borked with OutOfMemoryError, the data nodes seemed to have survived. Master (each): 16GB of which ES\_JAVA\_OPTS="-Xms7g -Xmx7g" Ingest (each): 4GB of which ES\_JAVA\_OPTS="-Xms3g -Xmx3g" Coordinator (each): 64GB of which ES\_JAVA\_OPTS="-Xms30g -Xmx30g" Data (each): 64GB of which ES\_JAVA\_OPTS="-Xms30g -Xmx30g" All nodes running ubuntu JDK openjdk version "1.8.0\_162" OpenJDK Runtime Environment (build 1.8.0\_162-8u162-b12-0ubuntu0.16.04.2-b12) OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode) All nodes have MAX\_OPEN\_FILES=65536 MAX\_LOCKED\_MEMORY=unlimited MAX\_MAP\_COUNT=262144 Mast node before the crash: ![34%20AM](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bf65a23a9b9a901c9dafaea3bea483ad38ea0005.png) Ingest node before the crash: ![18%20AM](https://us1.discourse-cdn.com/elastic/original/3X/8/a/8a613685abdc3fc69f2cc0a24284a46dd072c919.png) Master log: [https://www.dropbox.com/s/vro7yls5mmfmu0u/master.log?dl=0](https://www.dropbox.com/s/vro7yls5mmfmu0u/master.log?dl=0) Ingest Log: [https://www.dropbox.com/s/6trmcp5r0beulxa/ingest.log?dl=0](https://www.dropbox.com/s/6trmcp5r0beulxa/ingest.log?dl=0) ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [May 9, 2018, 4:15pm UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/2 "2018-05-09T16:15:41Z")
The fact that master and ingest nodes have the problem but not data nodes suggests that the issue might be related to the size of your cluster state. Maybe you have many indexes / shards / fields? What is the size of the output of `GET /_cluster/state`? ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [May 9, 2018, 5:43pm UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/3 "2018-05-09T17:43:52Z")
@jpountz Hi, 800 indexes and 8000 shards give or take. The state is about 720KB I should also add it's daily indexes, but most are small. Out of the 800... - 60 are between 1 to 3 million documents. - 100 are between 100K to 900K documents. - The rest are bellow 100K documents. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 10, 2018, 6:11am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/4 "2018-05-10T06:11:00Z")
You have too many shards, look to use `_shrink` on older ones and reduce the count or switch to weekly/monthly indices. ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [May 10, 2018, 6:37am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/5 "2018-05-10T06:37:37Z")
And if I want to keep that many say daily but maximum a year? Do I just increase the master ram to 30gb or add more nodes? Can I have older indexes as monthly and the newer ones as daily? How will the date math work if we can do this? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 10, 2018, 7:15am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/6 "2018-05-10T07:15:28Z")
Given your index size, keeping a year's worth of indices (plus that last month) around isn't going to be worth keeping daily with that shard count. ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [May 10, 2018, 7:33am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/7 "2018-05-10T07:33:41Z")
So can I take old ones make them monthly and new ones daily? Will Kibana date math work on both? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 10, 2018, 7:36am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/8 "2018-05-10T07:36:01Z")
Switching from daily to monthly requires reindexing, so it is better if you switch to monthly indices for all data. Kibana does not use date math based on index names any longer, so that is not a problem. ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [May 10, 2018, 8:01am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/9 "2018-05-10T08:01:08Z")
Ok. Thanks. ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [May 10, 2018, 8:06am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/10 "2018-05-10T08:06:09Z")
Ok I will reindex but I have to phase it obviously. So I will do older ones first to monthly and then the newer ones eventually. ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [May 10, 2018, 8:14am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/11 "2018-05-10T08:14:01Z")
On monthly indexes we cannot do daily backups though can we? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 10, 2018, 8:49am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/12 "2018-05-10T08:49:21Z")
Sure you can. ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [May 10, 2018, 12:39pm UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/13 "2018-05-10T12:39:28Z")
@warkolm Cool so reading the docs... Snapshots only store the changed files correct? So if a monthly index has NOT changed in 31 days and we took 31 snapshots the snapshot repo would remain the same size and NOT have grown right? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 11, 2018, 5:12am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/14 "2018-05-11T05:12:59Z")
> [@javadevmtl](#): > > Cool so reading the docs... Snapshots only store the changed files correct? So if a monthly index has NOT changed in 31 days and we took 31 snapshots the snapshot repo would remain the same size and NOT have grown right? More than likely. You can reduce the chance by running a forge merge once they have been written to. ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [May 14, 2018, 4:44pm UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/15 "2018-05-14T16:44:44Z")
Hi, so far it seems stable, thanks! I thought people were running with much more indexes? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 15, 2018, 8:11am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/16 "2018-05-15T08:11:59Z")
Forget about the number of indices, it's the shards that matter. ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [May 15, 2018, 3:04pm UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/17 "2018-05-15T15:04:32Z")
Ok cool gotcha. So, 1 index with 5 shards is the same as 5 indexes with 1 shard each. And each shard is a Lucene index which takes up X amount of resources. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 16, 2018, 4:47am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/18 "2018-05-16T04:47:04Z")
> [@javadevmtl](#): > > Ok cool gotcha. So, 1 index with 5 shards is the same as 5 indexes with 1 shard each. And each shard is a Lucene index which takes up X amount of resources. Exactly. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 13, 2018, 4:47am UTC](https://discuss.elastic.co/t/outofmemoryerror-on-adequately-sized-cluster/131192/19 "2018-06-13T04:47:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.