we want to schedule backup of elasticsearch indexes at regular interval and store the backup to s3 repo.
For this we have configured a service account and created isra role and associated this role to the service account.
In the deployment yaml of elastisearch i'm installing the repository-s3 plugin and creating a symlink to $AWS_WEB_IDENTITY_TOKEN_FILE. Below is the code snipet of yaml file
podTemplate:
spec:
serviceAccountName: cad-phr-elastic-sa
initContainers:
- name: install-plugins
command:
- sh
- -c
- |
bin/elasticsearch-plugin install --batch repository-s3
- |
- mkdir -p "/usr/share/elasticsearch/config/repository-s3"; ln -s $AWS_WEB_IDENTITY_TOKEN_FILE
"/usr/share/elasticsearch/config/repository-s3/aws-web-identity-token-file"
After deploying with above changes when we try to connect to s3 it gives us access denied error.
we tried a workaround by manually setting the AWS access keys within the elasticsearch pod and settting allow_insecure_settings to enable repository-s3. After this setting this manually we are able to connect to s3.
Reference link - https://discuss.elastic.co/t/elasticsearch-s3-snapshot/275639/5
But for scheduled backup we cannot use this workaround as the access keys expire after sometime and this may lead to failure of backups.
Please suggest on how to fix this access denied error.
Let us know if anymore details are required.
Reference link for AWS IAM roles for service accounts (IRSA)
https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-snapshots.html
### Steps to Reproduce
Create service account and IRSA policy to read data from s3 and associate this serviceaccount to elasticsearc pod
Deploy eleasticsearch by updating the yml file to install below reposistory
podTemplate:
spec:
serviceAccountName: cad-phr-elastic-sa
initContainers:
- name: install-plugins
command:
- sh
- -c
- |
bin/elasticsearch-plugin install --batch repository-s3
- |
- mkdir -p "/usr/share/elasticsearch/config/repository-s3"; ln -s $AWS_WEB_IDENTITY_TOKEN_FILE
"/usr/share/elasticsearch/config/repository-s3/aws-web-identity-token-file"
In kibana console run command to check the if you are able to connect to s3
PUT /_snapshot/elastic-data-repository-dst
{
"type": "s3",
"settings": {
"bucket": "ind-dev-s3-elastic-data-migration",
"role_arn" : "update_s3_role_arn_here",
"region":"ap-south-1",
"endpoint": "update_s3_endpoint"
}
}
we have also tried with higher version of elasticsearch 8.13.2 and we still get same access denied error.