# Working set memory High **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-alerting\ **Created:** [March 12, 2024, 8:58am UTC](https://discuss.elastic.co/t/working-set-memory-high/355215 "2024-03-12T08:58:10Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Kesavan\_raina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kesavan_raina/32/124748_2.png) [@Kesavan\_raina](https://discuss.elastic.co/u/Kesavan_raina)\ **Post date:** [March 12, 2024, 8:58am UTC](https://discuss.elastic.co/t/working-set-memory-high/355215/1 "2024-03-12T08:58:10Z")
Hi all, I'm running an Elasticsearch cluster with just one node JVM configuration uses the default values with ```auto -Xms 2g -Xmx 2g ``` but when I checked the memory usage I saw that the `working set memory` value is up to 25gb. This could be really critical, so why seems that Elastic doesn't respect the configuration? thanks in advance ```auto [2024-02-07T08:55:54,816][WARN][o.e.i.c.IndicesClusterStateService] [erp_master_ny1sv-vmutil02] [[rawsenttraf][0]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [rawsenttraf][0]: Recovery failed on {erp_master_ny1sv-vmutil02}{g5-Z4JNHSS-VPiDzDZlLrg}{4p1YULUTS6aJBuX4UDxouw}{10.32.49.55}{10.32.49.55:9301}{hostname=ny1sv-vmutil02, erp=true, location=F:\ManageEngine\Log360\Exchange Reporter Plus\bin\..\elasticsearch} at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2139) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.4.jar:6.5.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:430) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1645) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2135) ~[elasticsearch-6.5.4.jar:6.5.4] ... 4 more Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to open reader on writer at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:600) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:218) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:168) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2266) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2248) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1413) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1370) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:425) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1645) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2135) ~[elasticsearch-6.5.4.jar:6.5.4] ... 4 more Caused by: java.io.IOException: The paging file is too small for this operation to complete: MMapIndexInput(path="F:\ManageEngine\Log360\Exchange Reporter Plus\elasticsearch\data\nodes\0\indices\IqvR_kGsT8Sl52V6yZp-Fw\0\index\_6mj.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 1073741824 bytes. Windows is unfortunately very limited on virtual address space. If your index size is several hundred Gigabytes, consider changing to Linux. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html] at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[?:1.8.0_282] at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:924) ~[?:1.8.0_282] at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:267) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:242) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.(CompressingStoredFieldsReader.java:150) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:126) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:83) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:524) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:585) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:218) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:168) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2266) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2248) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1413) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1370) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:425) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1645) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2135) ~[elasticsearch-6.5.4.jar:6.5.4] ... 4 more [2024-02-07T08:55:54,847][WARN][o.e.c.r.a.AllocationService] [erp_master_ny1sv-vmutil02] failing shard [failed shard, shard [rawrcvdtraf][0], node[g5-Z4JNHSS-VPiDzDZlLrg], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=4GJxm7MFRVKOiadW5EBTyw], unassigned_info[[reason=ALLOCATION_FAILED], at[2024-02-07T13:55:52.829Z], failed_attempts[1], delayed=false, details[failed shard on node [g5-Z4JNHSS-VPiDzDZlLrg]: failed recovery, failure RecoveryFailedException[[rawrcvdtraf][0]: Recovery failed on {erp_master_ny1sv-vmutil02}{g5-Z4JNHSS-VPiDzDZlLrg}{4p1YULUTS6aJBuX4UDxouw}{10.32.49.55}{10.32.49.55:9301}{hostname=ny1sv-vmutil02, erp=true, location=F:\ManageEngine\Log360\Exchange Reporter Plus\bin\..\elasticsearch}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: IOException[The paging file is too small for this operation to complete: MMapIndexInput(path="F:\ManageEngine\Log360\Exchange Reporter Plus\elasticsearch\data\nodes\0\indices\446jrqMVQi2FVSdooV_ZVg\0\index\_46g.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 1073741824 bytes. Windows is unfortunately very limited on virtual address space. If your index size is several hundred Gigabytes, consider changing to Linux. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]]; ], allocation_status[no_valid_shard_copy]], message [failed recovery], failure [RecoveryFailedException[[rawrcvdtraf][0]: Recovery failed on {erp_master_ny1sv-vmutil02}{g5-Z4JNHSS-VPiDzDZlLrg}{4p1YULUTS6aJBuX4UDxouw}{10.32.49.55}{10.32.49.55:9301}{hostname=ny1sv-vmutil02, erp=true, location=F:\ManageEngine\Log360\Exchange Reporter Plus\bin\..\elasticsearch}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: IOException[The paging file is too small for this operation to complete: MMapIndexInput(path="F:\ManageEngine\Log360\Exchange Reporter Plus\elasticsearch\data\nodes\0\indices\446jrqMVQi2FVSdooV_ZVg\0\index\_46g.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 1073741824 bytes. Windows is unfortunately very limited on virtual address space. If your index size is several hundred Gigabytes, consider changing to Linux. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]]; ], markAsStale [true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [rawrcvdtraf][0]: Recovery failed on {erp_master_ny1sv-vmutil02}{g5-Z4JNHSS-VPiDzDZlLrg}{4p1YULUTS6aJBuX4UDxouw}{10.32.49.55}{10.32.49.55:9301}{hostname=ny1sv-vmutil02, erp=true, location=F:\ManageEngine\Log360\Exchange Reporter Plus\bin\..\elasticsearch} at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2139) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.5.4.jar:6.5.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:430) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1645) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2135) ~[elasticsearch-6.5.4.jar:6.5.4] ... 4 more Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to open reader on writer at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:600) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:218) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:168) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2266) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2248) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1413) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1370) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:425) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1645) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2135) ~[elasticsearch-6.5.4.jar:6.5.4] ... 4 more Caused by: java.io.IOException: The paging file is too small for this operation to complete: MMapIndexInput(path="F:\ManageEngine\Log360\Exchange Reporter Plus\elasticsearch\data\nodes\0\indices\446jrqMVQi2FVSdooV_ZVg\0\index\_46g.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 1073741824 bytes. Windows is unfortunately very limited on virtual address space. If your index size is several hundred Gigabytes, consider changing to Linux. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html] at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[?:1.8.0_282] at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:924) ~[?:1.8.0_282] at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:267) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:242) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.(CompressingStoredFieldsReader.java:150) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:126) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:83) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:524) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:585) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:218) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:168) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2266) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2248) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1413) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1370) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:425) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1645) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2135) ~[elasticsearch-6.5.4.jar:6.5.4] ... 4 more [2024-02-07T08:55:54,941][WARN][o.e.c.r.a.AllocationService] [erp_master_ny1sv-vmutil02] failing shard [failed shard, shard [rawsenttraf][0], node[g5-Z4JNHSS-VPiDzDZlLrg], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=xuVK7QwoREqHfxBrD87UNw], unassigned_info[[reason=ALLOCATION_FAILED], at[2024-02-07T13:55:52.720Z], failed_attempts[1], delayed=false, details[failed shard on node [g5-Z4JNHSS-VPiDzDZlLrg]: failed recovery, failure RecoveryFailedException[[rawsenttraf][0]: Recovery failed on {erp_master_ny1sv-vmutil02}{g5-Z4JNHSS-VPiDzDZlLrg}{4p1YULUTS6aJBuX4UDxouw}{10.32.49.55}{10.32.49.55:9301}{hostname=ny1sv-vmutil02, erp=true, location=F:\ManageEngine\Log360\Exchange Reporter Plus\bin\..\elasticsearch}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: IOException[The paging file is too small for this operation to complete: MMapIndexInput(path="F:\ManageEngine\Log360\Exchange Reporter Plus\elasticsearch\data\nodes\0\indices\IqvR_kGsT8Sl52V6yZp-Fw\0\index\_i4j.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 1073741824 bytes. Windows is unfortunately very limited on virtual address space. If your index size is several hundred Gigabytes, consider changing to Linux. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]]; ], allocation_status[no_valid_shard_copy]], message [failed recovery], failure [RecoveryFailedException[[rawsenttraf][0]: Recovery failed on {erp_master_ny1sv-vmutil02}{g5-Z4JNHSS-VPiDzDZlLrg}{4p1YULUTS6aJBuX4UDxouw}{10.32.49.55}{10.32.49.55:9301}{hostname=ny1sv-vmutil02, erp=true, location=F:\ManageEngine\Log360\Exchange Reporter Plus\bin\..\elasticsearch}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: IOException[The paging file is too small for this operation to complete: MMapIndexInput(path="F:\ManageEngine\Log360\Exchange Reporter Plus\elasticsearch\data\nodes\0\indices\IqvR_kGsT8Sl52V6yZp-Fw\0\index\_6mj.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 1073741824 bytes. Windows is unfortunately very limited on virtual address space. If your index size is several hundred Gigabytes, consider changing to Linux. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]]; ], markAsStale [true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [rawsenttraf][0]: Recovery failed on {erp_master_ny1sv-vmutil02}{g5-Z4JNHSS-VPiDzDZlLrg}{4p1YULUTS6aJBuX4UDxouw}{10.32.49.55}{10.32.49.55:9301}{hostname=ny1sv-vmutil02, erp=true, location=F:\ManageEngine\Log360\Exchange Reporter Plus\bin\..\elasticsearch} at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2139) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.5.4.jar:6.5.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:430) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1645) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2135) ~[elasticsearch-6.5.4.jar:6.5.4] ... 4 more Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to open reader on writer at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:600) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:218) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:168) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2266) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2248) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1413) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1370) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:425) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1645) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2135) ~[elasticsearch-6.5.4.jar:6.5.4] ... 4 more Caused by: java.io.IOException: The paging file is too small for this operation to complete: MMapIndexInput(path="F:\ManageEngine\Log360\Exchange Reporter Plus\elasticsearch\data\nodes\0\indices\IqvR_kGsT8Sl52V6yZp-Fw\0\index\_6mj.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 1073741824 bytes. Windows is unfortunately very limited on virtual address space. If your index size is several hundred Gigabytes, consider changing to Linux. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html] at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[?:1.8.0_282] at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:924) ~[?:1.8.0_282] at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:267) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:242) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.(CompressingStoredFieldsReader.java:150) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:126) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:83) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:524) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:585) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:218) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:168) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2266) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2248) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1413) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1370) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:425) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1645) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2135) ~[elasticsearch-6.5.4.jar:6.5.4] ... 4 more ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 12, 2024, 10:32am UTC](https://discuss.elastic.co/t/working-set-memory-high/355215/2 "2024-03-12T10:32:52Z")
The memory settings you mentioned (2gb) is for the Java HEAP. Elasticsearch / Lucene are using also available RAM off heap. The guide says to disable swapping. See [Disable swapping | Elasticsearch Guide [8.12] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html) By the way, you are using a very old version which is not supported anymore. You should definitely upgrade. ---
**Author:** ![Kesavan\_raina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kesavan_raina/32/124748_2.png) [@Kesavan\_raina](https://discuss.elastic.co/u/Kesavan_raina)\ **Post date:** [March 12, 2024, 4:29pm UTC](https://discuss.elastic.co/t/working-set-memory-high/355215/3 "2024-03-12T16:29:53Z")
The paging file is too small for this operation to complete: MMapIndexInput(path="F:\ManageEngine\Log360\Exchange Reporter Plus\elasticsearch\data\nodes\0\indices\IqvR\_kGsT8Sl52V6yZp-Fw\0\index\_6mj.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 1073741824 bytes. Windows is unfortunately very limited on virtual address space. If your index size is several hundred Gigabytes, consider changing to Linux. More information: [The Generics Policeman Blog: Use Lucene’s MMapDirectory on 64bit platforms, please!](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html)] This Error Reason? ---
**Author:** ![Kesavan\_raina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kesavan_raina/32/124748_2.png) [@Kesavan\_raina](https://discuss.elastic.co/u/Kesavan_raina)\ **Post date:** [March 12, 2024, 4:43pm UTC](https://discuss.elastic.co/t/working-set-memory-high/355215/4 "2024-03-12T16:43:29Z")
Memory Configuration -Xms5120m, -Xmx5120m ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 12, 2024, 9:17pm UTC](https://discuss.elastic.co/t/working-set-memory-high/355215/5 "2024-03-12T21:17:00Z")
My only advice is to upgrade. You are using a very old jvm, a very old elasticsearch version (with an old Lucene version). I'm feeling that most of those "problems" have been solved. But anyway, you need to [Disable swapping | Elasticsearch Guide [8.12] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html) ---
**Author:** ![Kesavan\_raina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kesavan_raina/32/124748_2.png) [@Kesavan\_raina](https://discuss.elastic.co/u/Kesavan_raina)\ **Post date:** [March 14, 2024, 9:33pm UTC](https://discuss.elastic.co/t/working-set-memory-high/355215/6 "2024-03-14T21:33:49Z")
![image (1)](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3d14a10f17396fbc1a4037c2dd7a0affbe42aa55.png) Is this setting not affecting the Windows machine? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 15, 2024, 3:52am UTC](https://discuss.elastic.co/t/working-set-memory-high/355215/7 "2024-03-15T03:52:48Z")
Yes probably. I'm not a windows server sys admin so I can't tell. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 12, 2024, 3:53am UTC](https://discuss.elastic.co/t/working-set-memory-high/355215/8 "2024-04-12T03:53:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.