Elasticsearch "snapshot_restore_exception" "reason": snapshot does not exist" when snapshot status is SUCCESS

[arozaki (Andrea)]

I am using the Java/Groovy client for Elasticsearch and I am trying to restore an ES snapshot during an upgrade. I am using the Groovy code I have so far for setting up my old and new clients and making them create & restore snapshots:

    def oldElasticClient = new ElasticClient("old", oldElasticPort)
    oldElasticClient.cleanIndicesAndTemplates()
    def newElasticClient = new ElasticClient("new", newElasticPort)
    newElasticClient.cleanIndicesAndTemplates()
  
    try {
      oldElasticClient.refreshAll()
      oldElasticClient.createSnapshotRepository()
      newElasticClient.createSnapshotRepository()
      oldElasticClient.createSnapshot()
      newElasticClient.restoreSnapshot()
      oldElasticClient.deleteSnapshot()
       ....
    finally {
      oldElasticClient.close()
      newElasticClient.close()
      .... 
}

and these are the methods which execute the actual create and restoration of the snapshots:

def createSnapshotRepository() {
    log.info("Creating snapshot repository on ${name} Elasticsearch...");
    def settings = Settings.builder()
      .put(FsRepository.LOCATION_SETTING.getKey(), "/var/tmp")
      .put(FsRepository.COMPRESS_SETTING.getKey(), true)
      .build()
    client.snapshot().createRepository(
      new PutRepositoryRequest("myrepository").settings(settings).type(FsRepository.TYPE),
      RequestOptions.DEFAULT
    )
    log.info("Done creating snapshot repository on ${name} Elasticsearch!");
  }

  def createSnapshot(Boolean waitForCompletion = true) {
    log.info("Creating snapshot on ${name} Elasticsearch...");
    CreateSnapshotRequest createSnapshotRequest1 = new CreateSnapshotRequest("myrepository", "mysnapshot_1")
    createSnapshotRequest1.waitForCompletion(true)
    createSnapshotRequest1.includeGlobalState(true)
    CreateSnapshotResponse createSnapshotResponse = client.snapshot().create(createSnapshotRequest1, RequestOptions.DEFAULT)
    log.info("All Snapshots")
    GetSnapshotsRequest getSnapshotsRequest = new GetSnapshotsRequest("myrepository");
    GetSnapshotsResponse getSnapshotsResponse = client.snapshot().get(getSnapshotsRequest, RequestOptions.DEFAULT)
    log.info(getSnapshotsResponse.getProperties().toString())
  }

 def restoreSnapshot() {
    log.info("Restoring snapshot on ${name} Elasticsearch...");
    client.snapshot().restore(
      new RestoreSnapshotRequest("myrepository", "mysnapshot_1")
        .includeGlobalState(true)
        .waitForCompletion(true),
      RequestOptions.DEFAULT
    )
    log.info("Done restoring snapshot on ${name} Elasticsearch!");
  }

After checking the logs I saw that my snapshot has been created successfully like so:

snapshots:[SnapshotInfo{snapshot=_na_: mysnapshot_1/niEyiuCtRny_fON1aXTMLQ, state=SUCCESS .... ]

However, my restoring method exits with an error message indicating, that the snapshot doesnt exist, which is really confusing me.This is the exact error:

"error":{"root_cause":[{"type":"snapshot_restore_exception","reason":"[myrepository:mysnapshot_1] snapshot does not exist"}],"type":"snapshot_restore_exception","reason":"[myrepository:mysnapshot_1] snapshot does not exist"},"status":500}

Would anyone know how I could debug this further to see what could be wrong? I am kinda clueless what the error could be as everything seems right to me. Any help would be much appreciated.