Elasticsearch 9.2.1 snapshot visible in repository but restore fails with snapshot_missing_exception

Elastic Stack Elasticsearch
1

I have an Elasticsearch 9.2.1 cluster configured with Snapshot Lifecycle Management (SLM) that takes nightly snapshots to an S3 repository. The snapshots are successfully created, and I can see all snapshot metadata in the S3 bucket.

I created a new Elasticsearch 9.2.1 cluster and registered a repository pointing to the same S3 bucket. The repository registration succeeds, and I can see the snapshots:

GET _snapshot/nightly-snapshots-003/_all

I can also retrieve information about an individual snapshot:

GET _snapshot/nightly-snapshots-003/snapshot-2026-06-11-00-00-00-utc

However, when I try to restore the snapshot:

POST /_snapshot/nightly-snapshots-003/snapshot-2026-06-11-00-00-00-utc/_restore?pretty

I get :

"error": {"root_cause": [{"type": "snapshot_missing_exception","reason": "[nightly-snapshots-003:snapshot-2026-06-11-00-00-00-utc] is missing"}],"type": "snapshot_missing_exception","reason": "[nightly-snapshots-003:snapshot-2026-06-11-00-00-00-utc] is missing","caused_by": {"type": "no_such_file_exception","reason": "Blob object [] not found: The specified key does not exist. (Service: S3, Status Code: 404)"}},"status": 404}

see the same behavior with multiple snapshots.

The confusing part is that:

  • The snapshot appears in the repository.
  • GET _snapshot/<repo>/<snapshot> works.
  • Only the restore operation fails because Elasticsearch tries to read metadata files that it says do not exist in S3.

Has anyone seen a situation where snapshot metadata is visible but restore fails due to missing blob files? Are there additional repository verification or consistency checks I should run?

2

Welcome!

This looks like a bug indeed. I guess that just after the POST request, if you try again the GET _snapshot/nightly-snapshots-003/snapshot-2026-06-11-00-00-00-utc, it works, right?

Is that happening only with this snapshot? Or if you try an older snapshot it fails the same way?

Do you see any details in logs, like a stacktrace?

3

This is a very odd message. There should be a blob name in the [] brackets.

Yes, see Verify the repository integrity | Elasticsearch API documentation

4

yes when I try with GET to check everything works fine I see my snapshot there and the status says success. also I tried with other snapshots the same issue I faced. in the message I see this:

curl -X POST https://xxx/_snapshot/nightly-snapshots-003/snapshot-2026-06-11-00-00-00-utc-mivyqa0-sm2mw-ylrthncq/_restore?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "snapshot_missing_exception",
        "reason" : "[nightly-snapshots-003:snapshot-2026-06-11-00-00-00-utc-mivyqa0-sm2mw-ylrthncq/_QbSq5jYQ8CxPfyLcPZ-pA] is missing"
      }
    ],
    "type" : "snapshot_missing_exception",
    "reason" : "[nightly-snapshots-003:snapshot-2026-06-11-00-00-00-utc-mivyqa0-sm2mw-ylrthncq/_QbSq5jYQ8CxPfyLcPZ-pA] is missing",
    "caused_by" : {
      "type" : "no_such_file_exception",
      "reason" : "Blob object [backup/indices/CRlya0ZTSaew9OTZlxmyAw/meta-iV_W4J0BJXbGAlFnZ1Wg.dat] not found: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: xxx, Extended Request ID: xx+xxxx+xxxx/xxx) (SDK Attempt Count: 1)"
    }
  },
  "status" : 404
}

also when I get the snapshot:

curl _snapshot/nightly-snapshots-003/snapshot-2026-06-11-00-00-00-utc-mivyqa0-sm2mw-ylrthncq?pretty"

{
  "snapshots" : [
    {
      "snapshot" : "snapshot-2026-06-11-00-00-00-utc-mivyqa0-sm2mw-ylrthncq",
      "uuid" : "_QbSq5jYQ8CxPfyLcPZ-pA",
      "repository" : "snapshots-003",
      "version_id" : 9039001,
      "version" : "9.2.0-9.2.1",
      "indices" : [
        ".ds-.slm-history-7-2026.05.13-000004",
        ".ds-.slm-history-7-2026.05.06-000003",
        ".ds-.slm-history-7-2026.06.10-000008",
        ".ds-.slm-history-7-2026.05.20-000005",
        ".ds-.slm-history-7-2026.04.29-000002",
        ".ds-ilm-history-7-2026.05.30-000004",
        ".ds-ilm-history-7-2026.04.30-000003",
        ".ds-.slm-history-7-2026.05.27-000006",
        ".ds-ilm-history-7-2026.04.07-000002",
        "folder",
        "agreement_templates",
        ".ds-.slm-history-7-2026.06.03-000007",
        "agreements",
        ".ds-ilm-history-7-2026.03.31-000001",
        ".ds-ilm-history-7-2026.06.06-000005",
        ".security-7",
        ".ds-.slm-history-7-2026.04.22-000001",
        ".ds-.logs-elasticsearch.deprecation-default-2026.04.30-000002",
        "address_book_person"
      ],
      "data_streams" : [
        ".logs-elasticsearch.deprecation-default",
        "ilm-history-7",
        ".slm-history-7"
      ],
      "include_global_state" : true,
      "metadata" : {
        "policy" : "snapshot-003"
      },
      "state" : "SUCCESS",
      "start_time" : "2026-06-10T23:59:59.859Z",
      "start_time_in_millis" : 1781135999859,
      "end_time" : "2026-06-11T00:09:43.588Z",
      "end_time_in_millis" : 1781136583588,
      "duration_in_millis" : 583729,
      "failures" : [ ],
      "shards" : {
        "total" : 30,
        "failed" : 0,
        "successful" : 30
      },
      "feature_states" : [
        {
          "feature_name" : "security",
          "indices" : [
            ".security-7"
          ]
        }
      ]
    }
  ],
  "total" : 1,
  "remaining" : 0
}

for any snapshots that I'm trying it's trying to find a x.dat file that doesn't exist. I also checked it manually in S3 bucket I couldn't find it.

5

sorry my bad I missed that part of the message:

curl -X POST https://xxx/_snapshot/nightly-snapshots-003/snapshot-2026-06-11-00-00-00-utc-mivyqa0-sm2mw-ylrthncq/_restore?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "snapshot_missing_exception",
        "reason" : "[nightly-snapshots-003:snapshot-2026-06-11-00-00-00-utc-mivyqa0-sm2mw-ylrthncq/_QbSq5jYQ8CxPfyLcPZ-pA] is missing"
      }
    ],
    "type" : "snapshot_missing_exception",
    "reason" : "[nightly-snapshots-003:snapshot-2026-06-11-00-00-00-utc-mivyqa0-sm2mw-ylrthncq/_QbSq5jYQ8CxPfyLcPZ-pA] is missing",
    "caused_by" : {
      "type" : "no_such_file_exception",
      "reason" : "Blob object [backup/indices/CRlya0ZTSaew9OTZlxmyAw/meta-iV_W4J0BJXbGAlFnZ1Wg.dat] not found: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: xxx, Extended Request ID: xx+xxxx+xxxx/xxx) (SDK Attempt Count: 1)"
    }
  },
  "status" : 404
}
6

Ok so the snapshot is still present in the object that lists all the available snapshots, but it's been damaged and this blob, on which it depends, is missing. That's usually because something else has removed it from the repository, maybe with a lifecycle policy that expires old objects.

7

I have an SLM policy that retains snapshots for 30 days. On the other hand, I also have an S3 lifecycle policy that deletes objects older than 30 days. Do you mean that I shouldn't have the S3 lifecycle policy and should instead let SLM manage snapshot retention in S3?

8

Yes, you cannot have anything else besides Elasticcsearch changing the data on snapshot buckets, you cannot have an S3 lifecycle policy deleting objects as this would delete the objects related to snapshots.

That's the issue, required files were deleted by the S3 policy.

9

Yep that'd do it. This will destroy your repository contents. See these docs:

You may use an S3 Lifecycle Policy to adjust the storage class of existing objects in your repository, but you must not transition objects to an unsupported class such as the Glacier classes, and you must not expire objects. If you use a Glacier storage class, or another unsupported storage class, or object expiry, then you may permanently lose access to your repository contents.