Elasticsearch 9.1.3 - Unable to Recover or Delete Protected System Indices .secrets-inference and .security-7 After Multiple Recovery Attempts

[Shubham_Khodpe]

Hello Team,

I am seeking assistance with an issue involving two protected system indices in our Elasticsearch 9.1.3 cluster:

• .secrets-inference
• .security-7

Environment

• Elasticsearch Version: 9.1.3
• Installation Type: TAR installation
• Multi-node cluster
• SSL enabled for transport communication
• Kibana 9.1.3
• Windows servers

Issue Description

We have been troubleshooting cluster health issues related to the above system indices. We have attempted multiple recovery and deletion procedures, but neither index can be recovered or removed.

The indices are protected system indices, and Elasticsearch prevents direct deletion even when using a superuser account.

What We Have Tried

1. Shard Recovery Attempts

Executed multiple times:

POST /_cluster/reroute?retry_failed=true

Checked shard allocation using:

GET /_cluster/allocation/explain

Reviewed cluster health and shard status repeatedly.

2. Close and Reopen Index

Attempted:

POST /.secrets-inference/_close
POST /.secrets-inference/_open

POST /.security-7/_close
POST /.security-7/_open

Followed by:

POST /_cluster/reroute?retry_failed=true

No improvement.

3. Node Restarts

Performed rolling restarts of Elasticsearch nodes.

No improvement.

4. Allocation Attempts

Attempted to allocate an empty primary shard:

POST /_cluster/reroute
{
"commands": [
{
"allocate_empty_primary": {
"index": ".secrets-inference",
"shard": 0,
"node": "data-24",
"accept_data_loss": true
}
}
]
}

Received:

{
"error": {
"type": "illegal_argument_exception",
"reason": "[allocate_empty_primary] primary [.secrets-inference][0] is already assigned"
}
}

This suggests Elasticsearch believes the primary shard is already assigned.

5. Deletion Attempts

Tried deleting through:

• Kibana Index Management
• Dev Tools
• curl API
• Superuser account

Example:

curl -k -u user:password -X DELETE "https://:9200/.secrets-inference"

Response:

{
"error": {
"type": "illegal_argument_exception",
"reason": "Indices [.secrets-inference] use and access is reserved for system operations"
},
"status": 400
}

The same protection applies to ".security-7".

6. Security Privileges

Created a dedicated user with:

• superuser role
• custom role with:
  • all cluster privileges
  • all index privileges
  • allow_restricted_indices: true

Despite this, deletion is still blocked.

Current Situation

• ".secrets-inference" cannot be recovered or deleted.
• ".security-7" is also affected.
• Elasticsearch treats both as protected system indices.
• Recovery actions have not resolved the issue.
• Deletion actions are blocked by system index protection.
• We want to avoid unsupported actions that may damage cluster security or metadata.

Questions

1. Has anyone successfully recovered a problematic ".secrets-inference" index in Elasticsearch 9.x?
2. What is the recommended recovery procedure when a protected system index becomes unhealthy?
3. Is there a supported method to rebuild or recreate ".secrets-inference"?
4. What are the supported recovery options for ".security-7"?
5. Is there any Elastic-supported procedure to remove or repair these indices when standard recovery methods fail?
6. Are there any additional diagnostics that should be collected to identify the root cause?

Any guidance, recovery procedure, or references to official documentation would be greatly appreciated.

Thank you.

[ropc (Romain Chanu)]

Hi @Shubham_Khodpe

We have been troubleshooting cluster health issues related to the above system indices. We have attempted multiple recovery and deletion procedures, but neither index can be recovered or removed.

May I know what happened to these two system indices? Did you experience any data loss?

If I assume a potential data loss, the only supported way to recover data is to use an Elasticsearch snapshot (c.f Snapshot and restore | Elastic Docs).

In version 8.x onwards, the only way to recover system indices is through the feature states (c.f Restore a snapshot | Elastic Docs)

If you do have a valid Elasticsearch snapshot, then restoring the inference_plugin and security feature states is the only way to restore these two system indices:

You could run a similar API:

POST _snapshot/your_repository_name/your_snapshot_name/_restore
{
  "feature_states": ["inference_plugin","security"],
  "include_global_state": false,
  "indices": "-*"
}

If you have an existing Support subscription with us, do make sure to open a Support ticket as it will be easier for our Support team to assist you more efficiently. If not, we will try our best to assist you here.

Thank you.

[Shubham_Khodpe]

Thank you for your response.

There was no known data loss in the cluster. The issue started after one of our data nodes was rebooted. Since then, the cluster health has remained yellow due to unassigned replica shards associated with the following system indices:

• ".secrets-inference"
• ".security-7"

The primary shards appear to be assigned and operational, and the cluster continues to function normally. However, the replica shards for these system indices remain unassigned.

We have already attempted the following without success:

• "POST /_cluster/reroute?retry_failed=true"
• Close and reopen the indices
• Rolling restart of Elasticsearch nodes
• Allocation explain analysis
• Allocation commands such as "allocate_empty_primary" (which returned that the primary shard is already assigned)
• Verification of cluster allocation settings and disk space
• Attempts to delete and recreate the indices (blocked because they are protected system indices)

At this point, our primary goal is to recover or reassign the replica shards so that the cluster can return to a green state.

Could you please advise:

1. What is the recommended recovery procedure when only replica shards of protected system indices remain unassigned?
2. Are there any supported methods to force regeneration of the replica shards for ".secrets-inference" and ".security-7"?
3. Are there any known issues in Elasticsearch 9.1.3 related to replica allocation for these system indices after a data node restart?

Unfortunately, we do not have a snapshot available, but since there is no known data loss and the primary shards are available, we are mainly looking for guidance on recovering the replica shards and restoring cluster health to green.

Thank you for your assistance.

[ropc (Romain Chanu)]

Hi @Shubham_Khodpe

Since then, the cluster health has remained yellow due to unassigned replica shards associated with the following system indices

Alright, if we are only talking about unassigned replica shards, we need to understand why they cannot be allocated. Note that if primary shards are already allocated - you should not try to tamper with them (e.g allocate_empty_primary ) as this will for sure cause data loss.

Are you able to share the following API results?

GET _cluster/health
GET _cat/nodes?v
GET _cat/allocation?v
GET _cluster/allocation/explain
GET _cluster/settings
GET .security-7/_settings

Feel free to obfuscate sensitive information (hostname, IP addresses) if needed but I will need most of the information otherwise for further troubleshooting.

Something must be preventing Elasticsearch from allocating the replica shards so we need to determine what it is (e.g disk usage issues, cluster settings, impossible allocation rules, etc.)

[Shubham_Khodpe]

Hi Romain,

Please find required details for your reference.

GET .security-7/_settings
#! this request accesses system indices: [.security-7], but in a future major version, direct access to system indices will be prevented by default
{
".security-7": {
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_hot,data_content"
}
}
},
"hidden": "true",
"number_of_shards": "1",
"auto_expand_replicas": "0-1",
"provided_name": ".security-7",
"format": "6",
"creation_date": "1771076350695",
"analysis": {
"filter": {
"email": {
"type": "pattern_capture",
"preserve_original": "true",
"patterns": [
"([^@]+)",
"""(\p{L}+)""",
"""(\d+)""",
"@(.+)"
]
}
},
"analyzer": {
"email": {
"filter": [
"email",
"lowercase",
"unique"
],
"tokenizer": "uax_url_email"
}
}
},
"priority": "1000",
"number_of_replicas": "1",
"uuid": "i1zYqbUBRwaSrS7RA76CCg",
"version": {
"created": "9033000"
}
}
}
}
}

GET _cluster/allocation/explain

{
"note": "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API. See Explain the shard allocations | Elasticsearch API documentation (v9) for more information.",
"index": ".security-7",
"shard": 0,
"primary": false,
"current_state": "unassigned",
"unassigned_info": {
"reason": "ALLOCATION_FAILED",
"at": "2026-06-08T12:48:56.668Z",
"failed_allocation_attempts": 5,
"details": """failed shard on node [hZhVpAyuQue0JnU7MVsEcQ]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [.security-7][0]: Recovery failed from {data-38}{KAdKV6zrRY2e5k8-j4NlXw}{g9CVtkF6RNe-tJJBXZlHTQ}{data-38}{10.30.0.38}{10.30.0.38:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} into {data-25}{hZhVpAyuQue0JnU7MVsEcQ}{CqGfXCXGQRyAH3R1seZuKw}{data-25}{10.30.4.25}{10.30.4.25:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0} (failed to prepare shard for recovery)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:296)
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:202)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:207)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:360)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.ActionListenerImplementations$NotifyOnceActionListener.onFailure(ActionListenerImplementations.java:390)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:415)
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:327)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:219)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:177)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:438)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:767)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
at java.lang.Thread.run(Thread.java:1447)
Caused by: java.nio.file.AccessDeniedException: E:\elastic-data\indices\i1zYqbUBRwaSrS7RA76CCg\0\index\write.lock
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:118)
at java.nio.channels.FileChannel.open(FileChannel.java:301)
at java.nio.channels.FileChannel.open(FileChannel.java:353)
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:112)
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:678)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$9(PeerRecoveryTargetService.java:419)
at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:105)
at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:403)
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:323)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:219)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:177)
at org.elasticsearch.action.support.SubscribableListener.lambda$andThenApply$2(SubscribableListener.java:537)
at org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
at org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:135)
at org.elasticsearch.action.support.SubscribableListener.andThenApply(SubscribableListener.java:537)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:398)
... 6 more
""",
"last_allocation_status": "no_attempt"
},
"can_allocate": "no",
"allocate_explanation": "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.",
"node_allocation_decisions": [
{
"node_id": "1PV-iB06R9m7HOT-D1-LGA",
"node_name": "data-37",
"transport_address": "10.30.0.37:9300",
"node_attributes": {
"ml.config_version": "12.0.0",
"transform.config_version": "10.0.0",
"xpack.installed": "true"
},
"roles": [
"data"
],
"node_decision": "no",
"deciders": [
{
"decider": "max_retry",
"decision": "NO",
"explanation": """shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [POST /_cluster/reroute?retry_failed] to retry, and for more information, see [Diagnose unassigned shards | Elastic Docs] [unassigned_info[[reason=ALLOCATION_FAILED], at[2026-06-08T12:48:56.668Z], failed_attempts[5], failed_nodes[[hZhVpAyuQue0JnU7MVsEcQ, 1PV-iB06R9m7HOT-D1-LGA]], delayed=false, last_node[hZhVpAyuQue0JnU7MVsEcQ], details[failed shard on node [hZhVpAyuQue0JnU7MVsEcQ]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [.security-7][0]: Recovery failed from {data-38}{KAdKV6zrRY2e5k8-j4NlXw}{g9CVtkF6RNe-tJJBXZlHTQ}{data-38}{10.30.0.38}{10.30.0.38:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} into {data-25}{hZhVpAyuQue0JnU7MVsEcQ}{CqGfXCXGQRyAH3R1seZuKw}{data-25}{10.30.4.25}{10.30.4.25:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0} (failed to prepare shard for recovery)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:296)
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:202)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:207)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:360)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.ActionListenerImplementations$NotifyOnceActionListener.onFailure(ActionListenerImplementations.java:390)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:415)
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:327)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:219)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:177)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:438)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:767)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
at java.lang.Thread.run(Thread.java:1447)
Caused by: java.nio.file.AccessDeniedException: E:\elastic-data\indices\i1zYqbUBRwaSrS7RA76CCg\0\index\write.lock
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:118)
at java.nio.channels.FileChannel.open(FileChannel.java:301)
at java.nio.channels.FileChannel.open(FileChannel.java:353)
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:112)
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:678)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$9(PeerRecoveryTargetService.java:419)
at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:105)
at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:403)
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:323)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:219)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:177)
at org.elasticsearch.action.support.SubscribableListener.lambda$andThenApply$2(SubscribableListener.java:537)
at org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
at org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:135)
at org.elasticsearch.action.support.SubscribableListener.andThenApply(SubscribableListener.java:537)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:398)
... 6 more
], allocation_status[no_attempt]]]"""
}
]
},
{
"node_id": "KAdKV6zrRY2e5k8-j4NlXw",
"node_name": "data-38",
"transport_address": "10.30.0.38:9300",
"node_attributes": {
"ml.config_version": "12.0.0",
"transform.config_version": "10.0.0",
"xpack.installed": "true"
},
"roles": [
"data"
],
"node_decision": "no",
"deciders": [
{
"decider": "max_retry",
"decision": "NO",
"explanation": """shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [POST /_cluster/reroute?retry_failed] to retry, and for more information, see [Diagnose unassigned shards | Elastic Docs] [unassigned_info[[reason=ALLOCATION_FAILED], at[2026-06-08T12:48:56.668Z], failed_attempts[5], failed_nodes[[hZhVpAyuQue0JnU7MVsEcQ, 1PV-iB06R9m7HOT-D1-LGA]], delayed=false, last_node[hZhVpAyuQue0JnU7MVsEcQ], details[failed shard on node [hZhVpAyuQue0JnU7MVsEcQ]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [.security-7][0]: Recovery failed from {data-38}{KAdKV6zrRY2e5k8-j4NlXw}{g9CVtkF6RNe-tJJBXZlHTQ}{data-38}{10.30.0.38}{10.30.0.38:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} into {data-25}{hZhVpAyuQue0JnU7MVsEcQ}{CqGfXCXGQRyAH3R1seZuKw}{data-25}{10.30.4.25}{10.30.4.25:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0} (failed to prepare shard for recovery)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:296)
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:202)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:207)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:360)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.ActionListenerImplementations$NotifyOnceActionListener.onFailure(ActionListenerImplementations.java:390)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:415)
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:327)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:219)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:177)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:438)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:767)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
at java.lang.Thread.run(Thread.java:1447)
Caused by: java.nio.file.AccessDeniedException: E:\elastic-data\indices\i1zYqbUBRwaSrS7RA76CCg\0\index\write.lock
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:118)
at java.nio.channels.FileChannel.open(FileChannel.java:301)
at java.nio.channels.FileChannel.open(FileChannel.java:353)
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:112)
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:678)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$9(PeerRecoveryTargetService.java:419)
at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:105)
at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:403)
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:323)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:219)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:177)
at org.elasticsearch.action.support.SubscribableListener.lambda$andThenApply$2(SubscribableListener.java:537)
at org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
at org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:135)
at org.elasticsearch.action.support.SubscribableListener.andThenApply(SubscribableListener.java:537)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:398)
... 6 more
], allocation_status[no_attempt]]]"""
},
{
"decider": "same_shard",
"decision": "NO",
"explanation": "a copy of this shard is already allocated to this node [[.security-7][0], node[KAdKV6zrRY2e5k8-j4NlXw], [P], s[STARTED], a[id=Is4lRdwARPq4MOnmsb7cRQ], failed_attempts[0]]"
}
]
},
{
"node_id": "bwbEoCmqTI6U_7l8JZAK_Q",
"node_name": "data-24",
"transport_address": "10.30.4.24:9300",
"node_attributes": {
"ml.config_version": "12.0.0",
"transform.config_version": "10.0.0",
"xpack.installed": "true"
},
"roles": [
"data"
],
"node_decision": "no",
"deciders": [
{
"decider": "max_retry",
"decision": "NO",
"explanation": """shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [POST /_cluster/reroute?retry_failed] to retry, and for more information, see [Diagnose unassigned shards | Elastic Docs] [unassigned_info[[reason=ALLOCATION_FAILED], at[2026-06-08T12:48:56.668Z], failed_attempts[5], failed_nodes[[hZhVpAyuQue0JnU7MVsEcQ, 1PV-iB06R9m7HOT-D1-LGA]], delayed=false, last_node[hZhVpAyuQue0JnU7MVsEcQ], details[failed shard on node [hZhVpAyuQue0JnU7MVsEcQ]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [.security-7][0]: Recovery failed from {data-38}{KAdKV6zrRY2e5k8-j4NlXw}{g9CVtkF6RNe-tJJBXZlHTQ}{data-38}{10.30.0.38}{10.30.0.38:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} into {data-25}{hZhVpAyuQue0JnU7MVsEcQ}{CqGfXCXGQRyAH3R1seZuKw}{data-25}{10.30.4.25}{10.30.4.25:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0} (failed to prepare shard for recovery)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:296)
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:202)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:207)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:360)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.ActionListenerImplementations$NotifyOnceActionListener.onFailure(ActionListenerImplementations.java:390)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:415)
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:327)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:219)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:177)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:438)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:767)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
at java.lang.Thread.run(Thread.java:1447)
Caused by: java.nio.file.AccessDeniedException: E:\elastic-data\indices\i1zYqbUBRwaSrS7RA76CCg\0\index\write.lock
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:118)
at java.nio.channels.FileChannel.open(FileChannel.java:301)
at java.nio.channels.FileChannel.open(FileChannel.java:353)
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:112)
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:678)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$9(PeerRecoveryTargetService.java:419)
at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:105)
at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:403)
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:323)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:219)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:177)
at org.elasticsearch.action.support.SubscribableListener.lambda$andThenApply$2(SubscribableListener.java:537)
at org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
at org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:135)
at org.elasticsearch.action.support.SubscribableListener.andThenApply(SubscribableListener.java:537)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:398)
... 6 more
], allocation_status[no_attempt]]]"""
}
]
},
{
"node_id": "hZhVpAyuQue0JnU7MVsEcQ",
"node_name": "data-25",
"transport_address": "10.30.4.25:9300",
"node_attributes": {
"ml.config_version": "12.0.0",
"xpack.installed": "true",
"transform.config_version": "10.0.0"
},
"roles": [
"data"
],
"node_decision": "no",
"deciders": [
{
"decider": "max_retry",
"decision": "NO",
"explanation": """shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [POST /_cluster/reroute?retry_failed] to retry, and for more information, see [Diagnose unassigned shards | Elastic Docs] [unassigned_info[[reason=ALLOCATION_FAILED], at[2026-06-08T12:48:56.668Z], failed_attempts[5], failed_nodes[[hZhVpAyuQue0JnU7MVsEcQ, 1PV-iB06R9m7HOT-D1-LGA]], delayed=false, last_node[hZhVpAyuQue0JnU7MVsEcQ], details[failed shard on node [hZhVpAyuQue0JnU7MVsEcQ]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [.security-7][0]: Recovery failed from {data-38}{KAdKV6zrRY2e5k8-j4NlXw}{g9CVtkF6RNe-tJJBXZlHTQ}{data-38}{10.30.0.38}{10.30.0.38:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} into {data-25}{hZhVpAyuQue0JnU7MVsEcQ}{CqGfXCXGQRyAH3R1seZuKw}{data-25}{10.30.4.25}{10.30.4.25:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0} (failed to prepare shard for recovery)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:296)
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:202)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:207)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:360)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.ActionListenerImplementations$NotifyOnceActionListener.onFailure(ActionListenerImplementations.java:390)
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:78)
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:89)
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32)
at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:415)
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:327)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:219)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:177)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:438)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:767)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
at java.lang.Thread.run(Thread.java:1447)
Caused by: java.nio.file.AccessDeniedException: E:\elastic-data\indices\i1zYqbUBRwaSrS7RA76CCg\0\index\write.lock
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:118)
at java.nio.channels.FileChannel.open(FileChannel.java:301)
at java.nio.channels.FileChannel.open(FileChannel.java:353)
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:112)
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:106)
at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:678)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$9(PeerRecoveryTargetService.java:419)
at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:105)
at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:403)
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:323)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:219)
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:177)
at org.elasticsearch.action.support.SubscribableListener.lambda$andThenApply$2(SubscribableListener.java:537)
at org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
at org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:135)
at org.elasticsearch.action.support.SubscribableListener.andThenApply(SubscribableListener.java:537)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:398)
... 6 more
], allocation_status[no_attempt]]]"""
}
]
}
]
}

[ropc (Romain Chanu)]

Hi @Shubham_Khodpe

If you check the results of the cluster allocation explain API, you will see that the replica shard allocation is failing due to an issue with data node data-25:

                    "details": """failed shard on node [hZhVpAyuQue0JnU7MVsEcQ]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [.security-7][0]: Recovery failed from {data-38}{KAdKV6zrRY2e5k8-j4NlXw}{g9CVtkF6RNe-tJJBXZlHTQ}{data-38}{10.30.0.38}{10.30.0.38:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} into {data-25}{hZhVpAyuQue0JnU7MVsEcQ}{CqGfXCXGQRyAH3R1seZuKw}{data-25}{10.30.4.25}{10.30.4.25:9300}{d}{9.1.3}{8000099-9033000}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0} (failed to prepare shard for recovery)
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java: 296)
...
...
Caused by: java.nio.file.AccessDeniedException: E:\elastic- data\indices\i1zYqbUBRwaSrS7RA76CCg\0\index\write.lock

As you can see, the Elasticsearch process running on node data-25 is unable to access the path E:\elastic-data\indices\i1zYqbUBRwaSrS7RA76CCg\0\index\write.lock due to permission issues (AccessDeniedException).

You will need to check the permissions for the user running the Elasticsearch process and ensure it has read/write access to the Elasticsearch data folder on that node.

Ensure as well that no other processes are trying to access/modify the Elasticsearch data folder/files - as this is not supported - c.f our documentation:

Don’t modify anything within the data directory or run processes that might interfere with its contents. If something other than Elasticsearch modifies the contents of the data directory, then Elasticsearch may fail, reporting corruption or other data inconsistencies, or may appear to work correctly having silently lost some of your data. Don’t attempt to take filesystem backups of the data directory; there is no supported way to restore such a backup. Instead, use Snapshot and restore to take backups safely. Don’t run virus scanners on the data directory. A virus scanner can prevent Elasticsearch from working correctly and may modify the contents of the data directory. The data directory contains no executables so a virus scan will only find false positives.