I am running ES (3 master, 3 data, and 3 ingest nodes) and Kibana on EKS using the ECK operator. I am trying to setup an S3 based snapshot repository but I receive a 401 Unauthorized error when trying to verify the repository in Kibana. I have configured everything based on the docs thus far.
- Created the S3 bucket with recommended policy.
- Created an IAM role that has access to the bucket.
- Manually modified the
defaultKubernetes service account which the Elasticsearch node pods use with the IAM role annotation. - Verified all the pods are assuming the IAM role which has access to the bucket.
- Verified access by connecting to a master node pod and uploaded/downloaded a file to the bucket using the awscli.
- Created the directory
/usr/share/elasticsearch/config/repository-s3and a symlink of$AWS_WEB_IDENTITY_TOKEN_FILEtoaws-web-identity-token-filein the same directory on all nodes. Did this after the pods were started. - Registered the snapshot repository in the Kibana UI using the correct bucket name and all the default settings.
When I click on the repository and then Verify repository, I get the following error. This happens with any bucket name I use. Any help would be appreciated.
{
"name": "ResponseError",
"meta": {
"body": {
"error": {
"root_cause": [
{
"type": "repository_verification_exception",
"reason": "[es-development-backups] path is not accessible on master node"
}
],
"type": "repository_verification_exception",
"reason": "[es-development-backups] path is not accessible on master node",
"caused_by": {
"type": "i_o_exception",
"reason": "Unable to upload object [tests-xTSezt2VQze-UcMp3Vv3rA/master.dat] using a single upload",
"caused_by": {
"type": "amazon_service_exception",
"reason": "amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null; Proxy: null)"
}
}
},
"status": 500
},
"statusCode": 500,
"headers": {
"x-opaque-id": "0aa85525-497f-47a6-b2bd-45777cfb9e45;kibana:application:management:",
"x-elastic-product": "Elasticsearch",
"content-type": "application/json;charset=utf-8",
"content-length": "594"
},
"meta": {
"context": null,
"request": {
"params": {
"method": "POST",
"path": "/_snapshot/es-development-backups/_verify",
"querystring": "",
"headers": {
"user-agent": "Kibana/8.5.1",
"x-elastic-product-origin": "kibana",
"authorization": "Basic REDACTED",
"x-opaque-id": "0aa85525-497f-47a6-b2bd-45777cfb9e45;kibana:application:management:",
"x-elastic-client-meta": "es=8.3.0p,js=16.18.1,t=8.2.0,hc=16.18.1",
"accept": "application/vnd.elasticsearch+json; compatible-with=8,text/plain"
}
},
"options": {
"opaqueId": "0aa85525-497f-47a6-b2bd-45777cfb9e45;kibana:application:management:",
"headers": {
"x-elastic-product-origin": "kibana",
"user-agent": "Kibana/8.5.1",
"authorization": "Basic REDACTED",
"x-opaque-id": "0aa85525-497f-47a6-b2bd-45777cfb9e45",
"x-elastic-client-meta": "es=8.3.0p,js=16.18.1,t=8.2.0,hc=16.18.1"
}
},
"id": 1
},
"name": "elasticsearch-js",
"connection": {
"url": "https://mycluster-dev-es-http.elastic-system.svc:9200/",
"id": "https://mycluster-dev-es-http.elastic-system.svc:9200/",
"headers": {},
"status": "alive"
},
"attempts": 0,
"aborted": false
},
"warnings": null
}
}
