# Elasticsearch 2.3.3 encountered outofmemory **URL:** \ **Category:** Elasticsearch\ **Created:** [July 25, 2016, 1:55am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255 "2016-07-25T01:55:34Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![tuankun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tuankun/32/8590_2.png) [@tuankun](https://discuss.elastic.co/u/tuankun)\ **Post date:** [July 25, 2016, 1:55am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/1 "2016-07-25T01:55:34Z")
Hi, we run 4 elk nodes in cluter using jre 1.8.0.77,logstash 2.3.2,elasticsearch 2.3.2 ,kibana 4.5.1.and there are totally about more than three hundred client server transfering linux log,windows event log and iis log to ELK cluster. the following is our architecture and H/W configuration: ![](https://us1.discourse-cdn.com/elastic/original/2X/0/0d8f0946edfb58ed82e6909c9b235370f4245da7.png) **but now we encoutered a critial error, logstash on ELK01 host is receiving and processing logs from client servers. every few days,the elasticsearch.log will log "OutOfMemory" exception,and quit from cluster,at the same time, I can not login OS through ssh remotely,so I have to force to reboot OS.** **Could anybody help to fix it? thanks.** [2016-07-25 01:22:35,370][WARN][index.engine] [elk04] [it\_p5sfcs\_iislog-2016.07.24][0] failed engine [refresh failed] java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:517) at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1931) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:455) at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:286) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:261) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:251) at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:137) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:154) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:672) at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:661) at org.elasticsearch.index.shard.IndexShard$EngineRefresher$1.run(IndexShard.java:1349) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 25, 2016, 1:59am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/2 "2016-07-25T01:59:21Z")
How much data in your cluster? ---
**Author:** ![tuankun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tuankun/32/8590_2.png) [@tuankun](https://discuss.elastic.co/u/tuankun)\ **Post date:** [July 25, 2016, 2:09am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/3 "2016-07-25T02:09:01Z")
@warkolm we configured number\_of\_replicas is 1, so totally is 50GB per day, ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 25, 2016, 2:10am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/4 "2016-07-25T02:10:23Z")
Ok, how many all together though? How many indices, how many shards? ---
**Author:** ![tuankun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tuankun/32/8590_2.png) [@tuankun](https://discuss.elastic.co/u/tuankun)\ **Post date:** [July 25, 2016, 2:16am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/5 "2016-07-25T02:16:16Z")
@warkolm every day will create 25 indices(each index owns 5 primary shards and 5 replicas shards),and keep past one month history indices. so now ,we have 787 indices and 7872 shards in our cluster. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 25, 2016, 2:17am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/6 "2016-07-25T02:17:41Z")
I'd say you are massively oversharded and that is creating heap pressure. ---
**Author:** ![tuankun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tuankun/32/8590_2.png) [@tuankun](https://discuss.elastic.co/u/tuankun)\ **Post date:** [July 25, 2016, 2:22am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/7 "2016-07-25T02:22:20Z")
@warkolm based our ELK cluster H/W,Could you have the advice about how many indices and shards? or How can I know the optimum amount values of indices and shards. ELK04 : HP DL580 Gen5 ELK01/02/03: HP DL380 Gen5 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 25, 2016, 3:07am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/8 "2016-07-25T03:07:09Z")
Aim for shard size \<50GB. ---
**Author:** ![tuankun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tuankun/32/8590_2.png) [@tuankun](https://discuss.elastic.co/u/tuankun)\ **Post date:** [July 25, 2016, 3:16am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/9 "2016-07-25T03:16:37Z")
@warkolm 1. is it right that one index(5 primary shards+5 replicas shards) size should be less than 500GB? 2. May I configure logstash output plugins to create index by week?,if yes,could you give me the date pattern? ---
**Author:** ![DiscussBuster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/discussbuster/32/11042_2.png) [@DiscussBuster](https://discuss.elastic.co/u/DiscussBuster)\ **Post date:** [July 25, 2016, 3:47am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/10 "2016-07-25T03:47:55Z")
> [@tuankun](#): > > 1. is it right that one index(5 primary shards+5 replicas shards) size should be less than 500GB? Sounds about right. > [@tuankun](#): > > 1. May I configure logstash output plugins to create index by week?,if yes,could you give me the date pattern? [Consult the documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index) ---
**Author:** ![tuankun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tuankun/32/8590_2.png) [@tuankun](https://discuss.elastic.co/u/tuankun)\ **Post date:** [July 25, 2016, 8:55am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/11 "2016-07-25T08:55:51Z")
@warkolm @DiscussBuster Really very thank you ,I will try to modify logstash configuration to create indices by week to reduce the amount of shards,then check the effect.😀 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:33pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-3-encountered-outofmemory/56255/12 "2017-07-05T22:33:13Z")