# Shard allocation failed with MMap warnings **URL:** \ **Category:** Elasticsearch\ **Created:** [October 1, 2016, 9:06am UTC](https://discuss.elastic.co/t/shard-allocation-failed-with-mmap-warnings/61983 "2016-10-01T09:06:25Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Shammi\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@Shammi\_Kumar](https://discuss.elastic.co/u/Shammi_Kumar)\ **Post date:** [October 1, 2016, 9:06am UTC](https://discuss.elastic.co/t/shard-allocation-failed-with-mmap-warnings/61983/1 "2016-10-01T09:06:25Z")
- I am using ES version 2.3.2 on 3 node cluster [12 core / 16 GB RAM per node] - I am writing per day index size with 100 GB (12 shards per index) daily. - After one day (reached 26 shards), I am getting following errors in ES logs. - One of the shard (out of 26) "state": "INITIALIZING" **How to overcome this issue? Its happening frequently. I have increased ulimit -m and ulimit -v "unlimited". and I have increased vm.max\_map\_count=262144 from default value**. [2016-10-01 14:12:55,269][WARN][indices.cluster] [hostname10] [[xxxx.yyyy.zzzz\_20161001][7]] marking and sending shard failed due to [failed recovery] [xxxx.yyyy.zzzz\_20161001][[xxxx.yyyy.zzzz\_20161001][7]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: IOException[Map failed: MMapIndexInput(path="/hdfs1/PERF-ATLAS/nodes/0/indices/xxxx.yyyy.zzzz\_20161001/7/index/\_2p3\_Lucene54\_0.dvd") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 65422441 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max\_map\_count'. More information: [http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html)]]; at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250) at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56) at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: [xxxx.yyyy.zzzz\_20161001][[xxxx.yyyy.zzzz\_20161001][7]] EngineCreationFailureException[failed to open reader on writer]; nested: IOException[Map failed: MMapIndexInput(path="/hdfs1/PERF-ATLAS/nodes/0/indices/xxxx.yyyy.zzzz\_20161001/7/index/\_2p3\_Lucene54\_0.dvd") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 65422441 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max\_map\_count'. More information: [http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html)]]; at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:295) at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:166) at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1515) at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1499) at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:972) at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:944) at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241) ... 5 more Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/hdfs1/PERF-ATLAS/nodes/0/indices/xxxx.yyyy.zzzz\_20161001/7/index/\_2p3\_Lucene54\_0.dvd") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 65422441 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max\_map\_count'. More information: [http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html)] at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:273) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:247) at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:186) at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89) at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.(Lucene54DocValuesProducer.java:132) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.(PerFieldDocValuesFormat.java:268) ---
**Author:** ![Shammi\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@Shammi\_Kumar](https://discuss.elastic.co/u/Shammi_Kumar)\ **Post date:** [October 1, 2016, 9:08am UTC](https://discuss.elastic.co/t/shard-allocation-failed-with-mmap-warnings/61983/2 "2016-10-01T09:08:07Z")
Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/hdfs1/PERF-ATLAS/nodes/0/indices/security.events.normalized\_20161001/7/index/\_2p3\_Lucene54\_0.dvd") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 65422441 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max\_map\_count'. More information: [http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html)] at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:273) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:247) at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:186) at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89) at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.(Lucene54DocValuesProducer.java:132) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.(PerFieldDocValuesFormat.java:268) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:358) at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:147) at org.apache.lucene.index.SegmentReader.(SegmentReader.java:81) at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:99) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:435) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:100) at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:283) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 1, 2016, 10:28pm UTC](https://discuss.elastic.co/t/shard-allocation-failed-with-mmap-warnings/61983/3 "2016-10-01T22:28:30Z")
> [@Shammi\_Kumar](#): > > per day index size with 100 GB (12 shards per index) daily That's way too many, you need 3 shards with 1 replica set. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [October 1, 2016, 11:01pm UTC](https://discuss.elastic.co/t/shard-allocation-failed-with-mmap-warnings/61983/4 "2016-10-01T23:01:03Z")
Is this being written to HDFS? ---
**Author:** ![Shammi\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@Shammi\_Kumar](https://discuss.elastic.co/u/Shammi_Kumar)\ **Post date:** [October 4, 2016, 7:18am UTC](https://discuss.elastic.co/t/shard-allocation-failed-with-mmap-warnings/61983/5 "2016-10-04T07:18:51Z")
Yes, I am writing to HDFS and to ES in parallel. ---
**Author:** ![Shammi\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@Shammi\_Kumar](https://discuss.elastic.co/u/Shammi_Kumar)\ **Post date:** [October 4, 2016, 7:24am UTC](https://discuss.elastic.co/t/shard-allocation-failed-with-mmap-warnings/61983/6 "2016-10-04T07:24:10Z")
Thank you. will try that option ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:15pm UTC](https://discuss.elastic.co/t/shard-allocation-failed-with-mmap-warnings/61983/7 "2017-07-05T22:15:17Z")