# Elasticsearch 7.7.1 shards getting unassigned **URL:** \ **Category:** Elasticsearch\ **Created:** [November 19, 2020, 2:55pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-1-shards-getting-unassigned/256006 "2020-11-19T14:55:18Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Deb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deb/32/46162_2.png) [@Deb](https://discuss.elastic.co/u/Deb)\ **Post date:** [November 19, 2020, 2:55pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-1-shards-getting-unassigned/256006/1 "2020-11-19T14:55:18Z")
We recently upgraded our elasticsearch cluster from 5.6.16 to 7.7.1. After that sometimes I am observing that few of the shards not getting assigned. My node stats is placed [here](https://jsonblob.com/b99c4071-2a74-11eb-ae20-7b4f84b0b1cb). The allocation explanation for an `unassigned` shard is like below ``` ubuntu@platform2:~$ curl -X GET "localhost:9200/_cluster/allocation/explain?pretty" -H 'Content-Type: application/json' -d' > { > "index": "denorm", > "shard": 14, > "primary": false > } > ' { "index" : "denorm", "shard" : 14, "primary" : false, "current_state" : "unassigned", "unassigned_info" : { "reason" : "ALLOCATION_FAILED", "at" : "2020-11-19T13:09:42.072Z", "failed_allocation_attempts" : 5, "details" : "failed shard on node [ocKks7zJT7OODhse-yveyg]: failed to perform indices:data/write/bulk[s] on replica [denorm][14], node[ocKks7zJT7OODhse-yveyg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=WpZNgGsuSoeyHE-Hg_HBSw], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-11-19T13:08:41.492Z], failed_attempts[4], failed_nodes[[ocKks7zJT7OODhse-yveyg, 0_00hk5IRcmgrHGYjpV1jA]], delayed=false, details[failed shard on node [0_00hk5IRcmgrHGYjpV1jA]: failed recovery, failure RecoveryFailedException[[denorm][14]: Recovery failed from {platform3}{9ltF-KXGRk-xMF_Ef1DAng}{hdd3KH53Sg6Us8Ow2rVY-A}{10.62.70.179}{10.62.70.179:9300}{dimr} into {platform2}{0_00hk5IRcmgrHGYjpV1jA}{0jbndos9TQq9s-DoSMNjgA}{10.62.70.178}{10.62.70.178:9300}{dimr}]; nested: RemoteTransportException[[platform3][10.62.70.179:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[platform2][10.62.70.178:9300][internal:index/shard/recovery/file_chunk]]; nested: CircuitBreakingException[[parent] Data too large, data for [] would be [4480371114/4.1gb], which is larger than the limit of [4476560998/4.1gb], real usage: [4479322256/4.1gb], new bytes reserved: [1048858/1mb], usages [request=0/0b, fielddata=1619769/1.5mb, in_flight_requests=2097692/2mb, accounting=119863546/114.3mb]]; ], allocation_status[no_attempt]], expected_shard_size[1168665714], failure RemoteTransportException[[platform1][10.62.70.177:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [] would be [4516429702/4.2gb], which is larger than the limit of [4476560998/4.1gb], real usage: [4516131576/4.2gb], new bytes reserved: [298126/291.1kb], usages [request=65648/64.1kb, fielddata=1412022/1.3mb, in_flight_requests=29783730/28.4mb, accounting=121642746/116mb]]; ", "last_allocation_status" : "no_attempt" }, "can_allocate" : "no", "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes", "node_allocation_decisions" : [ { "node_id" : "0_00hk5IRcmgrHGYjpV1jA", "node_name" : "platform2", "transport_address" : "10.62.70.178:9300", "node_decision" : "no", "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-11-19T13:09:42.072Z], failed_attempts[5], failed_nodes[[ocKks7zJT7OODhse-yveyg, 0_00hk5IRcmgrHGYjpV1jA]], delayed=false, details[failed shard on node [ocKks7zJT7OODhse-yveyg]: failed to perform indices:data/write/bulk[s] on replica [denorm][14], node[ocKks7zJT7OODhse-yveyg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=WpZNgGsuSoeyHE-Hg_HBSw], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-11-19T13:08:41.492Z], failed_attempts[4], failed_nodes[[ocKks7zJT7OODhse-yveyg, 0_00hk5IRcmgrHGYjpV1jA]], delayed=false, details[failed shard on node [0_00hk5IRcmgrHGYjpV1jA]: failed recovery, failure RecoveryFailedException[[denorm][14]: Recovery failed from {platform3}{9ltF-KXGRk-xMF_Ef1DAng}{hdd3KH53Sg6Us8Ow2rVY-A}{10.62.70.179}{10.62.70.179:9300}{dimr} into {platform2}{0_00hk5IRcmgrHGYjpV1jA}{0jbndos9TQq9s-DoSMNjgA}{10.62.70.178}{10.62.70.178:9300}{dimr}]; nested: RemoteTransportException[[platform3][10.62.70.179:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[platform2][10.62.70.178:9300][internal:index/shard/recovery/file_chunk]]; nested: CircuitBreakingException[[parent] Data too large, data for [] would be [4480371114/4.1gb], which is larger than the limit of [4476560998/4.1gb], real usage: [4479322256/4.1gb], new bytes reserved: [1048858/1mb], usages [request=0/0b, fielddata=1619769/1.5mb, in_flight_requests=2097692/2mb, accounting=119863546/114.3mb]]; ], allocation_status[no_attempt]], expected_shard_size[1168665714], failure RemoteTransportException[[platform1][10.62.70.177:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [] would be [4516429702/4.2gb], which is larger than the limit of [4476560998/4.1gb], real usage: [4516131576/4.2gb], new bytes reserved: [298126/291.1kb], usages [request=65648/64.1kb, fielddata=1412022/1.3mb, in_flight_requests=29783730/28.4mb, accounting=121642746/116mb]]; ], allocation_status[no_attempt]]]" } ] }, { "node_id" : "9ltF-KXGRk-xMF_Ef1DAng", "node_name" : "platform3", "transport_address" : "10.62.70.179:9300", "node_decision" : "no", "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-11-19T13:09:42.072Z], failed_attempts[5], failed_nodes[[ocKks7zJT7OODhse-yveyg, 0_00hk5IRcmgrHGYjpV1jA]], delayed=false, details[failed shard on node [ocKks7zJT7OODhse-yveyg]: failed to perform indices:data/write/bulk[s] on replica [denorm][14], node[ocKks7zJT7OODhse-yveyg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=WpZNgGsuSoeyHE-Hg_HBSw], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-11-19T13:08:41.492Z], failed_attempts[4], failed_nodes[[ocKks7zJT7OODhse-yveyg, 0_00hk5IRcmgrHGYjpV1jA]], delayed=false, details[failed shard on node [0_00hk5IRcmgrHGYjpV1jA]: failed recovery, failure RecoveryFailedException[[denorm][14]: Recovery failed from {platform3}{9ltF-KXGRk-xMF_Ef1DAng}{hdd3KH53Sg6Us8Ow2rVY-A}{10.62.70.179}{10.62.70.179:9300}{dimr} into {platform2}{0_00hk5IRcmgrHGYjpV1jA}{0jbndos9TQq9s-DoSMNjgA}{10.62.70.178}{10.62.70.178:9300}{dimr}]; nested: RemoteTransportException[[platform3][10.62.70.179:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[platform2][10.62.70.178:9300][internal:index/shard/recovery/file_chunk]]; nested: CircuitBreakingException[[parent] Data too large, data for [] would be [4480371114/4.1gb], which is larger than the limit of [4476560998/4.1gb], real usage: [4479322256/4.1gb], new bytes reserved: [1048858/1mb], usages [request=0/0b, fielddata=1619769/1.5mb, in_flight_requests=2097692/2mb, accounting=119863546/114.3mb]]; ], allocation_status[no_attempt]], expected_shard_size[1168665714], failure RemoteTransportException[[platform1][10.62.70.177:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [] would be [4516429702/4.2gb], which is larger than the limit of [4476560998/4.1gb], real usage: [4516131576/4.2gb], new bytes reserved: [298126/291.1kb], usages [request=65648/64.1kb, fielddata=1412022/1.3mb, in_flight_requests=29783730/28.4mb, accounting=121642746/116mb]]; ], allocation_status[no_attempt]]]" }, { "decider" : "same_shard", "decision" : "NO", "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[denorm][14], node[9ltF-KXGRk-xMF_Ef1DAng], [P], s[STARTED], a[id=SNyCoFUzSwaiIE4187Tfig]]" } ] }, { "node_id" : "ocKks7zJT7OODhse-yveyg", "node_name" : "platform1", "transport_address" : "10.62.70.177:9300", "node_decision" : "no", "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-11-19T13:09:42.072Z], failed_attempts[5], failed_nodes[[ocKks7zJT7OODhse-yveyg, 0_00hk5IRcmgrHGYjpV1jA]], delayed=false, details[failed shard on node [ocKks7zJT7OODhse-yveyg]: failed to perform indices:data/write/bulk[s] on replica [denorm][14], node[ocKks7zJT7OODhse-yveyg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=WpZNgGsuSoeyHE-Hg_HBSw], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-11-19T13:08:41.492Z], failed_attempts[4], failed_nodes[[ocKks7zJT7OODhse-yveyg, 0_00hk5IRcmgrHGYjpV1jA]], delayed=false, details[failed shard on node [0_00hk5IRcmgrHGYjpV1jA]: failed recovery, failure RecoveryFailedException[[denorm][14]: Recovery failed from {platform3}{9ltF-KXGRk-xMF_Ef1DAng}{hdd3KH53Sg6Us8Ow2rVY-A}{10.62.70.179}{10.62.70.179:9300}{dimr} into {platform2}{0_00hk5IRcmgrHGYjpV1jA}{0jbndos9TQq9s-DoSMNjgA}{10.62.70.178}{10.62.70.178:9300}{dimr}]; nested: RemoteTransportException[[platform3][10.62.70.179:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[platform2][10.62.70.178:9300][internal:index/shard/recovery/file_chunk]]; nested: CircuitBreakingException[[parent] Data too large, data for [] would be [4480371114/4.1gb], which is larger than the limit of [4476560998/4.1gb], real usage: [4479322256/4.1gb], new bytes reserved: [1048858/1mb], usages [request=0/0b, fielddata=1619769/1.5mb, in_flight_requests=2097692/2mb, accounting=119863546/114.3mb]]; ], allocation_status[no_attempt]], expected_shard_size[1168665714], failure RemoteTransportException[[platform1][10.62.70.177:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [] would be [4516429702/4.2gb], which is larger than the limit of [4476560998/4.1gb], real usage: [4516131576/4.2gb], new bytes reserved: [298126/291.1kb], usages [request=65648/64.1kb, fielddata=1412022/1.3mb, in_flight_requests=29783730/28.4mb, accounting=121642746/116mb]]; ], allocation_status[no_attempt]]]" } ] } ] } ``` As mentioned [here](https://discuss.elastic.co/t/data-too-large-data-for-transport-request-would-be-10554893106-9-8gb-which-is-larger-than-the-limit-of-10092838912-9-3gb-real-usage-10523239224-9-8gb-new-bytes-reserved-31653882-30-1mb/192215/11) in node stats I am observing that out of 4.3 GB heap only about ~85 MB is used for keeping in-memory data structures. As discussed [here](https://discuss.elastic.co/t/circuitbreakingexception-parent-data-too-large-in-es-7-x/192801) on setting `indices.breaker.total.use_real_memory: false` I am not seeing the `Data too large exception`. Can some one let me know how can I confirm if I am observing the same issue as discussed [here](https://github.com/elastic/elasticsearch/issues/44484)? Shard allocation is placed [here](https://gist.github.com/dmannaarkin/97d3c7b6e002b7a98ecdf8afce166860). ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 19, 2020, 4:12pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-1-shards-getting-unassigned/256006/2 "2020-11-19T16:12:34Z")
> [@Deb](#): > > `real usage: [4479322256/4.1gb]` Elasticsearch thinks it's using 4.1GB of heap; why do you think it's only using ~85MB? > [@Deb](#): > > Shard allocation is placed [here](https://gist.github.com/dmannaarkin/97d3c7b6e002b7a98ecdf8afce166860). That looks like far too many shards for such small indices. ---
**Author:** ![Deb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deb/32/46162_2.png) [@Deb](https://discuss.elastic.co/u/Deb)\ **Post date:** [November 19, 2020, 5:01pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-1-shards-getting-unassigned/256006/3 "2020-11-19T17:01:44Z")
Thanks @DavidTurner for replying. I misread the stats. I am seeing from the stats (that is posted in my question) ```auto "jvm": { "timestamp": 1605796569391, "uptime_in_millis": 9016168, "mem": { "heap_used_in_bytes": 3468059000, "heap_used_percent": 73, "heap_committed_in_bytes": 4712169472, "heap_max_in_bytes": 4712169472, "non_heap_used_in_bytes": 167725288, "non_heap_committed_in_bytes": 180129792, "pools": { "young": { "used_in_bytes": 900349776, "max_in_bytes": 907345920, "peak_used_in_bytes": 907345920, "peak_max_in_bytes": 907345920 }, "survivor": { "used_in_bytes": 112164136, "max_in_bytes": 113377280, "peak_used_in_bytes": 113377280, "peak_max_in_bytes": 113377280 }, "old": { "used_in_bytes": 2455545088, "max_in_bytes": 3691446272, "peak_used_in_bytes": 3691444384, "peak_max_in_bytes": 3691446272 } } }, ``` Just for my undestanding why it is showing `real usage: [4479322256/4.1gb]` in the circuit breaker log? Just to add same configuration with same load working fine with elasticsearch 5.6.16 we are seeing this issue only with our elasticsearch 7.7.1 clusters. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 19, 2020, 5:05pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-1-shards-getting-unassigned/256006/4 "2020-11-19T17:05:22Z")
> [@Deb](#): > > Just for my undestanding why it is showing `real usage: [4479322256/4.1gb]` in the circuit breaker log? That's the heap usage at the time of the failure. You might do better with 7.9 which has the benefit of [https://github.com/elastic/elasticsearch/pull/58674](https://github.com/elastic/elasticsearch/pull/58674). Apart from that my best suggestion is to reduce your shard count. ---
**Author:** ![Deb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deb/32/46162_2.png) [@Deb](https://discuss.elastic.co/u/Deb)\ **Post date:** [November 19, 2020, 5:15pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-1-shards-getting-unassigned/256006/5 "2020-11-19T17:15:23Z")
@DavidTurner One last query, we are using the default `ConcMarkSweepGC`. [https://github.com/elastic/elasticsearch/pull/58674](https://github.com/elastic/elasticsearch/pull/58674) will be helpful in case of `ConcMarkSweepGC` also or it is specific to G1GC ? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 19, 2020, 6:37pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-1-shards-getting-unassigned/256006/6 "2020-11-19T18:37:53Z")
Good point, the focus of that PR is indeed G1GC. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 17, 2020, 6:37pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-1-shards-getting-unassigned/256006/7 "2020-12-17T18:37:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.