Elaticsearch cluster back up is failing - "reason":"RepositoryMissingException"

Elastic Stack Elasticsearch
,
1

We've backup repository type as S3, and cluster is running as dockerized AWS EC2, recently snapshot process is failing with reason "reason":"RepositoryMissingException". This error is coming from one of the data node.

Checked POST /_SNAPSHOT//_VERIFY
Returned below error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[REPO_NAME] [[NODE_ID, 'RemoteTransportException[[NODE_NAME][NODE_IP:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[REPO_NAME] missing];']]"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[REPO_NAME] [[NODE_ID, 'RemoteTransportException[[NODE_NAME][NODE_IP:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[REPO_NAME] missing];']]"
  },
  "status" : 500
}

Verified inside the docker container, S3 plugin is installed.

Anyone faced this issue?

2

Is it a new node?

Are you using the client settings (access key and secret key) in elasticsearch.yml or on elasticsearch keystore?

If you are using the settings on keystores, check if the keystore for the node has the settings, you need to run elasticsearch-keystore list on that particular container.

3

Hi @leandrojmp
This data node is not a new node, S3 permissions are assigned from EC2 IAM role.

4

The error you are getting is probably related to permissions issues, but since I never used s3 repositories using EC2 IAM roles, so I can not help further.

Maybe someone else can give more insight.

Do you have anything else in Elasticsearch logs? Check the logs for this container to see if there anything that can give a hint of the issue.

5

Nothing from logs.
Thank you for your help.

6

What version are you using?

7

@DavidTurner ES version 5.6.16

8

5.X is very much EOL and no longer supported, you should be looking to upgrade as a matter of urgency.

9

@warkolm Sure.

Do you have any thoughts on the above error.

10

You need to upgrade, you are unlikely to get a resolution on such an old, unsupported version sorry to say.

11

Got it, Thank you @warkolm

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.