Repository-s3 plugin not synching snapshots

Elastic Stack Elasticsearch
1

Hi there, I have two elasticsearch clusters running on two separate kubernetes clusters. Both the elasticsearch clusters have the repository-s3 plugin installed.

    "2": {
        "name": "elasticsearch-client-575db98f7b-gd5gc",
        "component": "repository-s3",
        "version": "7.10.2"
    },
    "6": {
        "name": "elasticsearch-data-0",
        "component": "repository-s3",
        "version": "7.10.2"
    },
    "10": {
        "name": "elasticsearch-master-0",
        "component": "repository-s3",
        "version": "7.10.2"
    },

In one of the clusters, I am registering the repository in readonly mode, as follows:

PUT /_snapshot/backup_name_2_5k
{
  "type": "s3",
  "settings": {
      "base_path": "/backups/elasticsearch/backup_name_2_5k",
      "bucket": "backup",
      "region": "ap-south-1",
      "readonly": true
  }
}

And in another cluster in read-write mode as follows:

PUT /_snapshot/backup_name_2_5k
{
  "type": "s3",
  "settings": {
      "base_path": "/backups/elasticsearch/backup_name_2_5k",
      "bucket": "backup",
      "region": "ap-south-1"
  }
}

Both have the same repository name and the base path.

In the cluster where the repository is registered as read-write I am triggering the snapshot creation as follows:

PUT /_snapshot/backup_name_2_5k/backup_name_2_5k?wait_for_completion=true
{
    "indices": "vsa-*"
}

In the cluster where I created the snapshot, I see in the list the snapshot exists, and I confirmed in the s3 path the data has been written, but, in the cluster where I registered the repository as readonly the newly created snapshot is not showing up.

What could be the reason for this? I expect the snapshot to be listed in both the elasticsearch cluster as I use the same repository path with readonly flag in one of the clusters.

2

Welcome to our community! :smiley:

Elasticsearch 7.10 is EOL and no longer supported, you need to upgrade.

What is the output from GET _snapshot?

4

@warkolm the output of GET _snapshot is:

{
    "backup_name_2_5k": {
        "type": "s3",
        "settings": {
            "bucket": "backup",
            "base_path": "/backups/elasticsearch/backup_name_2_5k",
            "readonly": "true",
            "region": "ap-south-1"
        }
    }
}
6

What about from the other cluster?

7

For the other cluster it's

{
    "backup_name_2_5k": {
        "type": "s3",
        "settings": {
            "bucket": "backup",
            "region": "ap-south-1",
            "base_path": "/backups/elasticsearch/backup_name_2_5k"
        }
    }
}
8

Thanks, what about GET _cat/snapshots?v from both?

9

Both the clusters responded with:

{
    "error": {
        "root_cause": [
            {
                "type": "action_request_validation_exception",
                "reason": "Validation Failed: 1: repository is missing;"
            }
        ],
        "type": "action_request_validation_exception",
        "reason": "Validation Failed: 1: repository is missing;"
    },
    "status": 400
}
10

Can you run the 2nd command in Register a snapshot repository | Elasticsearch Guide [8.4] | Elastic?

11

Below is the output of POST /_snapshot/backup_name_2_5k/_verify

{
	"nodes": {
		"VSHLh5AxQX-8649VKaLi9w": {
			"name": "elasticsearch-data-0"
		},
		"xhBcoEP-RW6X2DSOhDtJXg": {
			"name": "elasticsearch-master-0"
		}
	}
}

After sending the above request, GET _cat/snapshots?v still returns the following:

{
    "error": {
        "root_cause": [
            {
                "type": "action_request_validation_exception",
                "reason": "Validation Failed: 1: repository is missing;"
            }
        ],
        "type": "action_request_validation_exception",
        "reason": "Validation Failed: 1: repository is missing;"
    },
    "status": 400
}
12

@warkolm do you know that issue might be happening?

13

can anyone from @elastic_team help here to understand what the issue could be?

14

Could you run:

GET /

On both clusters please?

15

I figured it out. For others, the issue was how you configure your repository, the base_path should not start with / (root path)

1 Like
16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.