NullPointerException when getting list of snapshots on S3


I'm using the repository-s3 plugin and after using it for some months I now cannot get the list of snapshots:

curl elasticsearch:9200/_snapshot/long_term/_all

Elasticsearch logs this:

[2017-07-25T12:01:47,038][WARN ][r.suppressed             ] path: /_snapshot/long_term/_all, params: {repository=long_term, snapshot=_all}
org.elasticsearch.transport.RemoteTransportException: [SVVyQPF][][cluster:admin/snapshot/get]
Caused by: java.lang.NullPointerException

I use curator to take the backups and after grabbing backups successfully it fails when it tries to delete old snapshots because that's when it requires a list too:

2017-07-25 11:53:02,191 ERROR     Failed to complete action: delete_snapshots.  <class 'curator.exceptions.FailedExecution'>: Unable to get snapshot information from repository: long_term.  Error: TransportError(500, 'null_pointer_exception', '[SVVyQPF][][cluster:admin/snapshot/get]')

I have a feeling this is due to some kind of timeout. I turned on debug logging and while I couldn't find a more specific reason this fails I noticed it made ~ 2K requests to S3 until it failed and it stopped at 90 seconds. Is this a configurable timeout?

Please help! I got no access to my backups!

That sounds very bad.

I think you should open an issue in elasticsearch.
Can you tell more about your version?

Just upgraded to 5.5.0.

And it appeared just after the upgrade?

Nope it started a few days before so it started while on 5.4.1.curling for the list of snapshots took increasingly long in the past but it still worked. Eventually I reached a curator timeout which I increased and it was working again. Now it seems to be breaking before it hits that timeout.

I see. AFAIK we don't set any 90s timeout when we are using the AWS SDK and defaults don't seem to set that as per

We are just setting the setSocketTimeout which is by default 50s.

And AFAIK it only set the timeout for a single API call.

May be it's a more global timeout that we have in snapshot/restore? @Igor_Motov Do you have an idea?

Opened an issue too:

1 Like

Can you check logs on around 2017-07-25T12:01:47 to see if there are any other messages there?

Added some more logs to the issue Regarding the time I am providing logs from a request initiated at 2017-07-25T12:26:30 because that's when I had DEBUG on.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.