NullPointerException when create snapshots on S3


(Pavlo) #1

Hello
We upgraded version of Elastisearch environment from 5.1 to 5.6.10 on cloud.elastic.co. Everything started working correctly except snapshorts. There is a problem with default snapshort repository. I see that ones in 30 minutes the process starts creating the snapshort, but it fails with NullPointerException error. Finally snapshort is removed from the storage.

[2018-07-16T19:59:09,406][WARN ][org.elasticsearch.snapshots.SnapshotsService] [found-snapshots:scheduled-1531770970-instance-0000000018/LohSOOovQWuSMFnK9fCATw] failed to finalize snapshot java.lang.NullPointerException: null 
at org.elasticsearch.repositories.RepositoryData.snapshotsToXContent(RepositoryData.java:357) ~[elasticsearch-5.6.10.jar:5.6.10] 
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeIndexGen(BlobStoreRepository.java:838) ~[elasticsearch-5.6.10.jar:5.6.10] 
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.finalizeSnapshot(BlobStoreRepository.java:568) ~[elasticsearch-5.6.10.jar:5.6.10] 
at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:978) [elasticsearch-5.6.10.jar:5.6.10] 
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10] 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144] 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144] 
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

We received the same error when we try to create snapshort by command from Kibana Dev Tools (or delete existing snapshort from storage).

{
  "error": {
    "root_cause": [
      {
        "type": "remote_transport_exception",
        "reason": "[instance-0000000018][172.17.0.10:19056][cluster:admin/snapshot/delete]"
      }
    ],
    "type": "null_pointer_exception",
    "reason": null
  },
  "status": 500
}

We have another S3 repository, mounted to this elasticsearch claster, that was created manually and we are able to create snapshort on this repository with command from Kibana Dev Tools


(Yannick Welsch) #2

The repository has become corrupted (more details here: https://github.com/elastic/elasticsearch/pull/30140)
This can happen if the lower-version cluster continues to write to the repository after the higher-version cluster has written to it.

Unfortunately, the only way to get back to a good state (where you can write into the repository) is to reset/clean the repository.


(Pavlo) #3

Thank you for answer. Could you advice who can I clean default S3 repository? Do I have an access to it? It was generated automatically during creation of elasticsearch deployment.


(Neil Brookes) #4

Hi Pavlo,

If this is for an Elastic Cloud instance, it is probably best to open a support case via the "Help" section of the Elastic Cloud interface.

Thanks
Neil


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.