# Ошибка "obtaining shard lock timed out after 5000ms"
**URL:** https://discuss.elastic.co/t/obtaining-shard-lock-timed-out-after-5000ms/252175
**Category:** Вопросы на русском языке
**Created:** [October 15, 2020, 10:44am UTC](https://discuss.elastic.co/t/obtaining-shard-lock-timed-out-after-5000ms/252175 "2020-10-15T10:44:53Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Sergei\_Frolov](https://discuss.elastic.co/u/Sergei_Frolov)
#### Post date: [October 15, 2020, 10:44am UTC](https://discuss.elastic.co/t/obtaining-shard-lock-timed-out-after-5000ms/252175/1 "2020-10-15T10:44:53Z")
Добрый день!
Сегодня обнаружил кластер в состоянии "yellow", один шард у системного `.kibana_task_manager_1` индекса был Unassigned.
посмотрев через
`GET /_cluster/allocation/explain`
обнаружил ошибку:
`obtaining shard lock timed out after 5000ms` (она касалась только одного индекса `.kibana_task_manager_1`)
тут же запустил
`POST /_cluster/reroute?retry_failed`
кластер позеленел.
В логах избранного мастера обнаружил по множеству индексов:
`[2020-10-15T06:14:57,913][ERROR][o.e.c.a.s.ShardStateAction] [es-tracker2b.node.ru] [event_log_tracker_2020_10-000001][3] unexpected failure while failing shard [shard id [[event_log_tracker_2020_10-000001][3]], allocation id [fYW_RhIGQtmaBLs_bmyxcQ], primary term [3], message [failed to perform indices:data/write/bulk[s] on replica [event_log_tracker_2020_10-000001][3], node[TBCq9WTdTQaAYTkufG9s2Q], [R], s[STARTED], a[id=fYW_RhIGQtmaBLs_bmyxcQ]], failure [RemoteTransportException[[es-tracker5b.node.ru][192.168.60.71:9300][indices:data/write/bulk[s][r]]]; nested: IllegalStateException[[event_log_tracker_2020_10-000001][3] operation primary term [3] is too old (current [4])]; ], markAsStale [true]] org.elasticsearch.cluster.action.shard.ShardStateAction$NoLongerPrimaryShardException: primary term [3] did not match current primary term [4] at org.elasticsearch.cluster.action.shard.ShardStateAction$ShardFailedClusterStateTaskExecutor.execute(ShardStateAction.java:365) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.1.jar:7.8.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] at java.lang.Thread.run(Thread.java:832) [?:?]`
там же, на мастере именно по индексу `.kibana_task_manager_1`:
`[2020-10-15T06:17:14,771][WARN][o.e.c.r.a.AllocationService] [es-tracker2b.node.ru] failing shard [failed shard, shard [.kibana_task_manager_1][0], node[E6wviW6_RweNcQZMQeYMhw], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=zEqj95CaR3qmGq0xht9jBw], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-10-15T03:17:04.328Z], failed_attempts[4], failed_nodes[[E6wviW6_RweNcQZMQeYMhw]], delayed=false, details[failed shard on node [E6wviW6_RweNcQZMQeYMhw]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[.kibana_task_manager_1][0]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation]]; ], allocation_status[no_attempt]], expected_shard_size[31336], message [failed to create shard], failure [IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[.kibana_task_manager_1][0]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation]]; ], markAsStale [true]] java.io.IOException: failed to obtain in-memory shard lock`
Версия 7.8.1, кластер - 6 нод, на каждой роли и master, и data
чтобы нода выпадала, такого не видел по графикам. Скажите, насколько проблема серьезна?
Есть ощущение что что-то глубже есть более серьезное и оно полыхнет сильнее, чем один неназначенный шард.
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [October 16, 2020, 7:04pm UTC](https://discuss.elastic.co/t/obtaining-shard-lock-timed-out-after-5000ms/252175/2 "2020-10-16T19:04:23Z")
Это признак нестабильности в кластере. Но сказать, чем была вызвана нестабильность сложно. Надо копаться в логах до того как кластер пожелтел.
---
### Author:  [@Sergei\_Frolov](https://discuss.elastic.co/u/Sergei_Frolov)
#### Post date: [October 16, 2020, 7:39pm UTC](https://discuss.elastic.co/t/obtaining-shard-lock-timed-out-after-5000ms/252175/3 "2020-10-16T19:39:25Z")
Спасибо за ответ, Игорь!
Возможно ли, что это было вызвано настройкой, которую я делаю всегда пока настраиваю кластер (сейчас все возвращено на дефолт, после того как вернул на дефолт, проблемы не наблюдаю):
```auto
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.node_concurrent_recoveries": "50",
"cluster.routing.allocation.cluster_concurrent_rebalance": "20",
"indices.recovery.max_bytes_per_sec": "2500mb"
}
}
```
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [October 16, 2020, 7:50pm UTC](https://discuss.elastic.co/t/obtaining-shard-lock-timed-out-after-5000ms/252175/4 "2020-10-16T19:50:00Z")
Теоретически - да, могло. Но оно ли вызвало-ли в этом случае сказать не могу.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 13, 2020, 7:50pm UTC](https://discuss.elastic.co/t/obtaining-shard-lock-timed-out-after-5000ms/252175/5 "2020-11-13T19:50:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.