Hi,
Currently we are following the guide here[1]. We can successfully assign proper service account(hence AWS iam role) to the es pods. If relevant here[2] you can see our redacted elasticsearch CRD file.
This correctly results in assigning the proper role to the es nodes(I can test if the role works simply by logging in to pod console & test some s3 copy operations.)
Next I refer to [1] to create the symlinks in the folder - here is a catch, "ES_PATH_CONF" is not set, hence we set it to " /usr/share/elasticsearch/config" & create a link to the AWS TOKEN as written in same documentation.
But still when we try to verify the repository connection status we get the error [3]
Any idea or guidance is appreciated, we are stuck.
Thanks!
Celal
[1]S3 repository | Elasticsearch Guide [8.5] | Elastic
[2]
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: <REDACTED>
spec:
version: <REDACTED>
image: <REDACTED>
auth:
fileRealm:
- secretName: <REDACTED>
- secretName: <REDACTED>
nodeSets:
- name: default
count: 2
podTemplate:
spec:
serviceAccount: <SA Name that binds to proper AWS Role>
serviceAccountName: <SA Name that binds to proper AWS Role>
imagePullSecrets:
- name: <REDACTED>
containers:
- name: elasticsearch
resources: <REDACTED>
volumeClaimTemplates:
- metadata:
name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: <REDACTED>
config:
node.store.allow_mmap: <REDACTED>
[3]
{
"name": "ResponseError",
"meta": {
"body": {
"error": {
"root_cause": [
{
"type": "repository_verification_exception",
"reason": "[repository-s3] path [<REDACTED>] is not accessible on master node"
}
],
"type": "repository_verification_exception",
"reason": "[repository-s3] path [<REDACTED>] is not accessible on master node",
"caused_by": {
"type": "i_o_exception",
"reason": "Unable to upload object [<REDACTED>/tests-hQo9BhOSQ5S1VGbd2mCXoA/master.dat] using a single upload",
"caused_by": {
"type": "amazon_s3_exception",
"reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: <REDACTED>; S3 Extended Request ID: <REDACTED>; Proxy: null)"
}
}
},
"status": 500
},
"statusCode": 500,
"headers": {
"x-opaque-id": "<REDACTED>;kibana:application:management:",
"x-elastic-product": "Elasticsearch",
"content-type": "application/json",
"content-length": "732"
},
"meta": {
"context": null,
"request": {
"params": {
"method": "POST",
"path": "/_snapshot/repository-s3/_verify",
"querystring": "",
"headers": {
"user-agent": "Kibana/8.9.0",
"x-elastic-product-origin": "kibana",
"authorization": "Basic <REDACTED>",
"x-opaque-id": "<REDACTED>;kibana:application:management:",
"x-elastic-client-meta": "es=8.8.0p,js=16.20.1,t=8.3.1,hc=16.20.1",
"accept": "application/vnd.elasticsearch+json; compatible-with=8,text/plain"
}
},
"options": {
"opaqueId": "<REDACTED>;kibana:application:management:",
"headers": {
"x-elastic-product-origin": "kibana",
"user-agent": "Kibana/8.9.0",
"authorization": "Basic <REDACTED>",
"x-opaque-id": "<REDACTED>",
"x-elastic-client-meta": "es=8.8.0p,js=16.20.1,t=8.3.1,hc=16.20.1"
}
},
"id": 1
},
"name": "elasticsearch-js",
"connection": {
"url": "<REDACTED>",
"id": "<REDACTED>",
"headers": {},
"status": "alive"
},
"attempts": 0,
"aborted": false
},
"warnings": null
}
}