NullPointerException when create snapshots on S3

Elastic Stack Elasticsearch
1

Hello
We upgraded version of Elastisearch environment from 5.1 to 5.6.10 on cloud.elastic.co. Everything started working correctly except snapshorts. There is a problem with default snapshort repository. I see that ones in 30 minutes the process starts creating the snapshort, but it fails with NullPointerException error. Finally snapshort is removed from the storage.

[2018-07-16T19:59:09,406][WARN ][org.elasticsearch.snapshots.SnapshotsService] [found-snapshots:scheduled-1531770970-instance-0000000018/LohSOOovQWuSMFnK9fCATw] failed to finalize snapshot java.lang.NullPointerException: null 
at org.elasticsearch.repositories.RepositoryData.snapshotsToXContent(RepositoryData.java:357) ~[elasticsearch-5.6.10.jar:5.6.10] 
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeIndexGen(BlobStoreRepository.java:838) ~[elasticsearch-5.6.10.jar:5.6.10] 
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.finalizeSnapshot(BlobStoreRepository.java:568) ~[elasticsearch-5.6.10.jar:5.6.10] 
at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:978) [elasticsearch-5.6.10.jar:5.6.10] 
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10] 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144] 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144] 
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

We received the same error when we try to create snapshort by command from Kibana Dev Tools (or delete existing snapshort from storage).

{
  "error": {
    "root_cause": [
      {
        "type": "remote_transport_exception",
        "reason": "[instance-0000000018][172.17.0.10:19056][cluster:admin/snapshot/delete]"
      }
    ],
    "type": "null_pointer_exception",
    "reason": null
  },
  "status": 500
}

We have another S3 repository, mounted to this elasticsearch claster, that was created manually and we are able to create snapshort on this repository with command from Kibana Dev Tools

2

The repository has become corrupted (more details here: https://github.com/elastic/elasticsearch/pull/30140)
This can happen if the lower-version cluster continues to write to the repository after the higher-version cluster has written to it.

Unfortunately, the only way to get back to a good state (where you can write into the repository) is to reset/clean the repository.

3

Thank you for answer. Could you advice who can I clean default S3 repository? Do I have an access to it? It was generated automatically during creation of elasticsearch deployment.

4

Hi Pavlo,

If this is for an Elastic Cloud instance, it is probably best to open a support case via the "Help" section of the Elastic Cloud interface.

Thanks
Neil

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.