# Shards UNASSIGNED and OOM in logs **URL:** \ **Category:** Elasticsearch\ **Created:** [November 3, 2014, 10:29am UTC](https://discuss.elastic.co/t/shards-unassigned-and-oom-in-logs/20538 "2014-11-03T10:29:02Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Matteo\_Cremolini](https://avatars.discourse-cdn.com/v4/letter/m/9e8a1a/32.png) [@Matteo\_Cremolini](https://discuss.elastic.co/u/Matteo_Cremolini)\ **Post date:** [November 3, 2014, 10:29am UTC](https://discuss.elastic.co/t/shards-unassigned-and-oom-in-logs/20538/1 "2014-11-03T10:29:02Z")
Hi, We are using Elasticsearch 1.3.2 on a 2 nodes cluster in a production environment. Currently only a few indexes contain about 1000 documents. Mapper-attachments plugin is used in model. Some shards in one cluster node (SERVER 1) are in the following states: { state: INITIALIZING primary: false node: KkuMLz0\_TKONN77uOoWE7A relocating\_node: null shard: 3 index: programma\_mare-trashcan } { state: UNASSIGNED primary: false node: null relocating\_node: null shard: 0 index: programma\_mare\_index\_initial } There are a lot of warning messages like the following: SERVER 1: org.elasticsearch.transport.RemoteTransportException: [inl-cdcl-ind2][inet[/10.73.193.51:9300]][index/shard/recovery/startRecovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [context][1] Phase[1] Execution failed at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1078) at org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636) at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135) at org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: [context][1] Failed to transfer [0] files with total size of [0b] at org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280) at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1074) ... 9 more Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174) at sun.nio.ch.IOUtil.read(IOUtil.java:195) at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:700) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:685) at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:176) at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342) at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) at org.apache.lucene.store.DataInput.readInt(DataInput.java:96) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:346) at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:457) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:907) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:753) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:453) at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:98) at org.elasticsearch.index.store.Store.readLastCommittedSegmentsInfo(Store.java:124) at org.elasticsearch.index.store.Store.access$300(Store.java:74) at org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:442) at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433) at org.elasticsearch.index.store.Store.getMetadata(Store.java:144) at org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145) ... 10 more SERVER 2: [2014-11-02 00:02:12,250][WARN][cluster.action.shard] [inl-cdcl-ind2] [context][1] received shard failed for [context][1], node[KkuMLz0\_TKONN77uOoWE7A], [R], s[INITIALIZING], indexUUID [4GkRH6dNR--kmx0FtXwcRA], reason [Failed to start shard, message [RecoveryFailedException[[context][1]: Recovery failed from [inl-cdcl-ind2][2eBj7ijRS82V8md2a78U-A][inl-cdcl-ind2][inet[/10.73.193.51:9300]] into [inl-cdcl-ind1][KkuMLz0\_TKONN77uOoWE7A][inl-cdcl-ind1][inet[/10.73.193.50:9300]]]; nested: RemoteTransportException[[inl-cdcl-ind2][inet[/10.73.193.51:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[context][1] Phase[1] Execution failed]; nested: RecoverFilesRecoveryException[[context][1] Failed to transfer [0] files with total size of [0b]]; nested: OutOfMemoryError[Direct buffer memory]; ]] ... [2014-11-01 00:02:28,251][WARN][cluster.action.shard] [inl-cdcl-ind2] [my-index][3] received shard failed for [my-index][3], node[KkuMLz0\_TKONN77uOoWE7A], [R], s[INITIALIZING], indexUUID [Y9nKzEPuRSGPsYhjisxYPQ], reason [master [inl-cdcl-ind2][2eBj7ijRS82V8md2a78U-A][inl-cdcl-ind2][inet[/10.73.193.51:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure] How can we identify the problem? Thanks, Regards Matteo Cremolini -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b6f7a5fa-7117-4e82-bf86-1b309d73b1e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6f7a5fa-7117-4e82-bf86-1b309d73b1e4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:52am UTC](https://discuss.elastic.co/t/shards-unassigned-and-oom-in-logs/20538/2 "2017-07-06T00:52:32Z")