# Cluster State Yellow: 2 shards initializing with multiple failed attempts: IllegalArgumentException \[ReleasableBytesStreamOutput cannot hold more than 2GB of data **URL:** \ **Category:** Elasticsearch\ **Created:** [May 22, 2023, 10:37am UTC](https://discuss.elastic.co/t/cluster-state-yellow-2-shards-initializing-with-multiple-failed-attempts-illegalargumentexception-releasablebytesstreamoutput-cannot-hold-more-than-2gb-of-data/334008 "2023-05-22T10:37:31Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Sarit\_Ghosh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarit_ghosh/32/110260_2.png) [@Sarit\_Ghosh](https://discuss.elastic.co/u/Sarit_Ghosh)\ **Post date:** [May 22, 2023, 10:37am UTC](https://discuss.elastic.co/t/cluster-state-yellow-2-shards-initializing-with-multiple-failed-attempts-illegalargumentexception-releasablebytesstreamoutput-cannot-hold-more-than-2gb-of-data/334008/1 "2023-05-22T10:37:31Z")
For about a week, we are seeing the following error and cluster state yellow. On checking the \_cluster/state we get this - Elastic Search Version - 7.17 (Please let me know if more data is needed) {"state":"INITIALIZING","primary":false,"node":"SU7gTPbgSOuXoIrYteGYWA","relocating\_node":null,"shard":2,"index":"","expected\_shard\_size\_in\_bytes":31061140722,"recovery\_source":{"type":"PEER"},"allocation\_id":{"id":"ufdEJ5FqRMqiN09EKM\_s6A"},"unassigned\_info":{"reason":"ALLOCATION\_FAILED","at":"2023-05-22T08:56:04.059Z", **"failed\_attempts":7** ,"failed\_nodes": ... IllegalArgumentException[ReleasableBytesStreamOutput cannot hold more than 2GB of data]; ] Full Stack Trace - ```auto IllegalArgumentException[ReleasableBytesStreamOutput cannot hold more than 2GB of data]; ], markAsStale [true]] org.elasticsearch.transport.SendRequestTransportException: [S**200000*][xx.xx.xx.xx:9300][indices:data/write/bulk[s][r]] at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:988) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:206) ~[?:?] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$300(SecurityServerTransportInterceptor.java:53) ~[?:?] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:167) ~[?:?] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:874) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:797) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1282) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:279) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:99) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:288) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.support.RetryableAction.run(RetryableAction.java:77) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:298) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:203) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:150) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:447) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:233) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:196) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:245) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:134) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:196) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.5.jar:7.17.5] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.lang.IllegalArgumentException: ReleasableBytesStreamOutput cannot hold more than 2GB of data at org.elasticsearch.common.io.stream.BytesStreamOutput.ensureCapacity(BytesStreamOutput.java:175) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.common.io.stream.BytesStreamOutput.writeBytes(BytesStreamOutput.java:85) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.common.io.stream.StreamOutput.write(StreamOutput.java:516) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.common.bytes.BytesArray.writeTo(BytesArray.java:122) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.common.io.stream.StreamOutput.writeBytesReference(StreamOutput.java:189) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.index.IndexRequest.writeBody(IndexRequest.java:747) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.index.IndexRequest.writeThin(IndexRequest.java:728) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.DocWriteRequest.writeDocumentRequestThin(DocWriteRequest.java:266) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.bulk.BulkItemRequest.writeThin(BulkItemRequest.java:115) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.bulk.BulkShardRequest.lambda$writeTo$3(BulkShardRequest.java:72) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.common.io.stream.StreamOutput.writeArray(StreamOutput.java:941) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.bulk.BulkShardRequest.writeTo(BulkShardRequest.java:69) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteShardRequest.writeTo(TransportReplicationAction.java:1387) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteReplicaRequest.writeTo(TransportReplicationAction.java:1459) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:75) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:180) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.transport.OutboundHandler.sendRequest(OutboundHandler.java:109) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:288) ~[elasticsearch-7.17.5.jar:7.17.5] at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:975) ~[elasticsearch-7.17.5.jar:7.17.5] ... 28 more ``` ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [May 24, 2023, 9:22am UTC](https://discuss.elastic.co/t/cluster-state-yellow-2-shards-initializing-with-multiple-failed-attempts-illegalargumentexception-releasablebytesstreamoutput-cannot-hold-more-than-2gb-of-data/334008/2 "2023-05-24T09:22:58Z")
> [@Sarit\_Ghosh](#): > > IllegalArgumentException[ReleasableBytesStreamOutput cannot hold more than 2GB of data]; ] Hi @Sarit_Ghosh, From the stack trace you have provided it looks like the returned result set is larger than 2G. This is larger than the amount that Elasticsearch can return in a single request. Have you been able to find out the query being run that is triggering this error? I would recommend having a look at [this related thread](https://discuss.elastic.co/t/discover-internal-server-error/193661/9) which does give some suggestions on diagnosing the problem further down. ---
**Author:** ![Sarit\_Ghosh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarit_ghosh/32/110260_2.png) [@Sarit\_Ghosh](https://discuss.elastic.co/u/Sarit_Ghosh)\ **Post date:** [May 24, 2023, 9:27am UTC](https://discuss.elastic.co/t/cluster-state-yellow-2-shards-initializing-with-multiple-failed-attempts-illegalargumentexception-releasablebytesstreamoutput-cannot-hold-more-than-2gb-of-data/334008/3 "2023-05-24T09:27:06Z")
Hi Carly. Thanks for looking into this. We often see cluster state yellow, and during these times if we check the \_cluster/state we see this error. This is happening frequently for a specific index where the cluster state is mostly yellow. Are you suggesting that this could be due to some query ? Could you please suggest few ways we can find out the query ? Or can we restrict the response size by some settings ? I will go through the link you have given. Thanks again. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 24, 2023, 9:40am UTC](https://discuss.elastic.co/t/cluster-state-yellow-2-shards-initializing-with-multiple-failed-attempts-illegalargumentexception-releasablebytesstreamoutput-cannot-hold-more-than-2gb-of-data/334008/4 "2023-05-24T09:40:23Z")
> [@carly.richmond](#): > > From the stack trace you have provided it looks like the returned result set is larger than 2G. Not quite, this exception is being thrown during indexing and not search. The solution would be to avoid creating such large bulk requests. ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [May 24, 2023, 9:40am UTC](https://discuss.elastic.co/t/cluster-state-yellow-2-shards-initializing-with-multiple-failed-attempts-illegalargumentexception-releasablebytesstreamoutput-cannot-hold-more-than-2gb-of-data/334008/5 "2023-05-24T09:40:46Z")
Thanks for the clarification @DavidTurner! ---
**Author:** ![Sarit\_Ghosh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarit_ghosh/32/110260_2.png) [@Sarit\_Ghosh](https://discuss.elastic.co/u/Sarit_Ghosh)\ **Post date:** [May 24, 2023, 9:43am UTC](https://discuss.elastic.co/t/cluster-state-yellow-2-shards-initializing-with-multiple-failed-attempts-illegalargumentexception-releasablebytesstreamoutput-cannot-hold-more-than-2gb-of-data/334008/6 "2023-05-24T09:43:51Z")
Thanks for all the responses. I will check the bulk requests happening to the index. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 21, 2023, 9:44am UTC](https://discuss.elastic.co/t/cluster-state-yellow-2-shards-initializing-with-multiple-failed-attempts-illegalargumentexception-releasablebytesstreamoutput-cannot-hold-more-than-2gb-of-data/334008/7 "2023-06-21T09:44:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.