Elasticsearch GCS snapshots RepositoryException

[navaneethan (Navaneethakrishnan)]

Hi Team,

We are suddenly facing the snapshot exception in GCS
with

[2023-06-05T11:40:34,219][WARN ][r.suppressed ] [###] path: /_snapshot/e#-repo/###_gcs_ss_2023-06-05-1130, params: {pretty=true, repository=###-###-repo, snapshot=###_gcs_ss_2023-06-05-1130}
org.elasticsearch.repositories.RepositoryException: [###-###-repo] could not read repository data from index blob

Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input in VALUE_STRING
at [Source: sun.nio.ch.ChannelInputStream@171ce462; line: 1, column: 14681217]

and we are not able to access the repo

{
"error": {
"root_cause": [
{
"type": "repository_exception",
"reason": "[###-repo] could not read repository data from index blob"
}
],
"type": "repository_exception",
"reason": "[###-repo] could not read repository data from index blob",
"caused_by": {
"type": "json_e_o_f_exception",
"reason": "Unexpected end-of-input in VALUE_STRING\n at [Source: sun.nio.ch.ChannelInputStream@768613c2; line: 1, column: 14681217]"
}
},
"status": 500
}

[warkolm (Mark Walkom)]

@DavidTurner sorry to ping you directly here but this seems right in your area of expertise.

[DavidTurner]

No problem Mark

What version are you running @navaneethan ?

[navaneethan (Navaneethakrishnan)]

We are using es 7.3 version but this is not related to the gcs bucket we are using for the snapshots

[DavidTurner]

Ok that version is very old, long past EOL. I remember we fixed an issue that would explain this a few years ago. You need to upgrade as a matter of urgency.