Executed reroute on today's index which was showing Yellow:
POST _cluster/reroute?retry_failed=true
On running GET (command below) on recovery process, I could see recovery taking place in terms of percentage.
GET _cat/recovery/prod_test_access-2020.09.03?v
After it completed 100%, health still shows as Yellow with one unassigned shard for the same (today's) index.
Shall I drop the replica for today's index and readd?
Why is it showing as not allocated?
Sorry. Where do you see it unallocated?
What does a _cluster/allocation/explain say on why it's unallocated?
Here is the output of allocation/explain:
{
"index" : "prod_test_access-2020.09.03",
"shard" : 0,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2020-09-03T14:03:38.063Z",
"failed_allocation_attempts" : 5,
"details" : "failed shard on node [2ULg0RFsSZiaUYm4SUpDOQ]: failed to perform indices:data/write/bulk[s] on replica [prod_test_access-2020.09.03][0], node[2ULg0RFsSZiaUYm4SUpDOQ], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=LT4uKYoPSBCYiUIm4xCibA], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-09-03T14:03:06.722Z], failed_attempts[4], failed_nodes[[2ULg0RFsSZiaUYm4SUpDOQ, y1s252NOR5amHRRjWpu5jA]], delayed=false, details[failed shard on node [y1s252NOR5amHRRjWpu5jA]: failed to perform indices:data/write/bulk[s] on replica [prod_test_access-2020.09.03][0], node[y1s252NOR5amHRRjWpu5jA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=6HZ6qmCTRsecCYZudceSDg], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-09-03T14:02:21.461Z], failed_attempts[3], failed_nodes[[2ULg0RFsSZiaUYm4SUpDOQ, y1s252NOR5amHRRjWpu5jA]], delayed=false, details[failed shard on node [2ULg0RFsSZiaUYm4SUpDOQ]: failed to perform indices:data/write/bulk[s] on replica [prod_test_access-2020.09.03][0], node[2ULg0RFsSZiaUYm4SUpDOQ], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=MNGK3GTQR2CWU2XtTm5NnQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-09-03T14:01:01.986Z], failed_attempts[2], failed_nodes[[2ULg0RFsSZiaUYm4SUpDOQ, y1s252NOR5amHRRjWpu5jA]], delayed=false, details[failed shard on node [y1s252NOR5amHRRjWpu5jA]: failed recovery, failure RecoveryFailedException[[prod_test_access-2020.09.03][0]: Recovery failed from {xxx-xxx-xxx}{acF0z4ZvQNy0lE5fykyf_Q}{0Jl-faJ_R26v8dlzzJ1OaQ}{xxx-xxx-xxx}{123-345-567:9300}{dil}{ml.machine_memory=67378692096, ml.max_open_jobs=20, xpack.installed=true} into {xxx-xxx-xxx}{y1s252NOR5amHRRjWpu5jA}{Sw0AaQSwQR6qJEVc5mHaqA}{xxx-xxx-xxx}{123-345-567:9300}{dil}{ml.machine_memory=67378692096, xpack.installed=true, ml.max_open_jobs=20}]; nested: RemoteTransportException[[xxx-xxx-xxx][123-345-567:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[xxx-xxx-xxx][123-345-567:9300][internal:index/shard/recovery/translog_ops]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [993359896/947.3mb], which is larger than the limit of [986061209/940.3mb], real usage: [992302552/946.3mb], new bytes reserved: [1057344/1mb], usages [request=0/0b, fielddata=18581/18.1kb, in_flight_requests=1057344/1mb, accounting=210380623/200.6mb]]; ], allocation_status[no_attempt]], expected_shard_size[40551605435], failure RemoteTransportException[[xxx-xxx-xxx][123-345-567:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [991720182/945.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [991595760/945.6mb], new bytes reserved: [124422/121.5kb], usages [request=0/0b, fielddata=16859/16.4kb, in_flight_requests=1495582/1.4mb, accounting=225438133/214.9mb]]; ], allocation_status[no_attempt]], expected_shard_size[34150060094], failure RemoteTransportException[[xxx-xxx-xxx][123-345-567:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [989038996/943.2mb], which is larger than the limit of [986061209/940.3mb], real usage: [988874824/943mb], new bytes reserved: [164172/160.3kb], usages [request=0/0b, fielddata=18581/18.1kb, in_flight_requests=1219938/1.1mb, accounting=205318354/195.8mb]]; ], allocation_status[no_attempt]], expected_shard_size[33902587303], failure RemoteTransportException[[xxx-xxx-xxx][123-345-567:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [986870246/941.1mb], which is larger than the limit of [986061209/940.3mb], real usage: [986791752/941mb], new bytes reserved: [78494/76.6kb], usages [request=0/0b, fielddata=16859/16.4kb, in_flight_requests=1137206/1mb, accounting=225693585/215.2mb]]; ",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "2ULg0RFsSZiaUYm4SUpDOQ",
"node_name" : "xxx-xxx-xxx",
"transport_address" : "123-345-567:9300",
"node_attributes" : {
"ml.machine_memory" : "67378692096",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "max_retry",
"decision" : "NO",
"explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-09-03T14:03:38.063Z], failed_attempts[5], failed_nodes[[2ULg0RFsSZiaUYm4SUpDOQ, y1s252NOR5amHRRjWpu5jA]], delayed=false, details[failed shard on node [2ULg0RFsSZiaUYm4SUpDOQ]: failed to perform indices:data/write/bulk[s] on replica [prod_test_access-2020.09.03][0], node[2ULg0RFsSZiaUYm4SUpDOQ], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=LT4uKYoPSBCYiUIm4xCibA], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-09-03T14:03:06.722Z], failed_attempts[4], failed_nodes[[2ULg0RFsSZiaUYm4SUpDOQ, y1s252NOR5amHRRjWpu5jA]], delayed=false, details[failed shard on node [y1s252NOR5amHRRjWpu5jA]: failed to perform indices:data/write/bulk[s] on replica [prod_test_access-2020.09.03][0], node[y1s252NOR5amHRRjWpu5jA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=6HZ6qmCTRsecCYZudceSDg], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-09-03T14:02:21.461Z], failed_attempts[3], failed_nodes[[2ULg0RFsSZiaUYm4SUpDOQ, y1s252NOR5amHRRjWpu5jA]], delayed=false, details[failed shard on node [2ULg0RFsSZiaUYm4SUpDOQ]: failed to perform indices:data/write/bulk[s] on replica [prod_test_access-2020.09.03][0], node[2ULg0RFsSZiaUYm4SUpDOQ], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=MNGK3GTQR2CWU2XtTm5NnQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-09-03T14:01:01.986Z], failed_attempts[2], failed_nodes[[2ULg0RFsSZiaUYm4SUpDOQ, y1s252NOR5amHRRjWpu5jA]], delayed=false, details[failed shard on node [y1s252NOR5amHRRjWpu5jA]: failed recovery, failure RecoveryFailedException[[prod_test_access-2020.09.03][0]: Recovery failed from {xxx-xxx-xxx}{acF0z4ZvQNy0lE5fykyf_Q}{0Jl-faJ_R26v8dlzzJ1OaQ}{xxx-xxx-xxx}{123-345-567:9300}{dil}{ml.machine_memory=67378692096, ml.max_open_jobs=20, xpack.installed=true} into {xxx-xxx-xxx}{y1s252NOR5amHRRjWpu5jA}{Sw0AaQSwQR6qJEVc5mHaqA}{xxx-xxx-xxx}{123-345-567:9300}{dil}{ml.machine_memory=67378692096, xpack.installed=true, ml.max_open_jobs=20}]; nested: RemoteTransportException[[xxx-xxx-xxx][123-345-567:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[xxx-xxx-xxx][123-345-567:9300][internal:index/shard/recovery/translog_ops]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [993359896/947.3mb], which is larger than the limit of [986061209/940.3mb], real usage: [992302552/946.3mb], new bytes reserved: [1057344/1mb], usages [request=0/0b, fielddata=18581/18.1kb, in_flight_requests=1057344/1mb, accounting=210380623/200.6mb]]; ], allocation_status[no_attempt]], expected_shard_size[40551605435], failure RemoteTransportException[[xxx-xxx-xxx][123-345-567:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [991720182/945.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [991595760/945.6mb], new bytes reserved: [124422/121.5kb], usages [request=0/0b, fielddata=16859/16.4kb, in_flight_requests=1495582/1.4mb, accounting=225438133/214.9mb]]; ], allocation_status[no_attempt]], expected_shard_size[34150060094], failure RemoteTransportException[[xxx-xxx-xxx][123-345-567:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [989038996/943.2mb], which is larger than the limit of [986061209/940.3mb], real usage: [988874824/943mb], new bytes reserved: [164172/160.3kb], usages [request=0/0b, fielddata=18581/18.1kb, in_flight_requests=1219938/1.1mb, accounting=205318354/195.8mb]]; ], allocation_status[no_attempt]], expected_shard_size[33902587303], failure RemoteTransportException[[xxx-xxx-xxx][123-345-567:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [986870246/941.1mb], which is larger than the limit of [986061209/940.3mb], real usage: [986791752/941mb], new bytes reserved: [78494/76.6kb], usages [request=0/0b, fielddata=16859/16.4kb, in_flight_requests=1137206/1mb, accounting=225693585/215.2mb]]; ], allocation_status[no_attempt]]]"
}
]
}
There is more but it is same as above.
Primarily CircuitBreakingException[[parent] Data too large
Today the unassigned number has risen to 4. See below
{
"cluster_name" : "elkcluster-prod",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 8,
"number_of_data_nodes" : 5,
"active_primary_shards" : 195,
"active_shards" : 386,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 4,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 98.97435897435898
}
Hi All,
I think I know where the issue is. "CircuitBreakingException[[parent] Data too large" error lead me to issues related to memory. I increased the heap size to just less than 50% of the physical RAM as advised in https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
Did _cluster/reroute?retry_failed=true one more time and the health is now back to Green with no unassigned shard.
Thanks @Warkolm for the useful tips as I got to learn new things in the process!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.