Setup snapshot for s3 plugin


I'm trying to setup the s3 plugin but I have an issue.
I don't how to format the secret for aws credentials for the operator.
Actually I tried that (with terraform):
resource "kubernetes_secret" "aws_credentials_datawarehouse" {
type = ""

  metadata {
    name = "datawarehouse-aws-credentials"

  data = {
    "s3.client.default.access_key" = "....."
    "s3.client.default.secret_key" = "....."

The operator log an error:
E1212 13:40:26.641312 1 reflector.go:126] pkg/mod/ Failed to list *v1beta1.Elasticsearch: v1beta1.ElasticsearchList.Items: []v1beta1.Elasticsearch: v1beta1.Elasticsearch.Spec: v1beta1.ElasticsearchSpec.SecureSettings: []v1beta1.SecretSource: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|ttings":["datawareho|..., bigger context ...|rageClassName":"standard"}}]}],"secureSettings":["datawarehouse-aws-credentials"],"updateStrategy":{|...

I tried to check in the github repository but I'm not sure of which format is expected inside the secret

The error looks more like a bug in the Elasticsearch resource itself.
Can you share your elasticsearch yaml manifest which references the secure settings secrets?

That error looks very similar to the one we sometimes see v1beta1 resources using the v1alpha1 format:

Thank you @Anya_Sabo , I forgot secureSettings is a list of map, in my case it was a list of string (the secret name).

Ok so the elasticsearch cluster take the secure settings but it seems it don't read or don't understand the secure settings.
The payload to create the snapshot:
"schedule": "0 0 * * * ?",
"name": "<hourly-snap-{now/d{yyyy.MM.dd.HH}}>",
"repository": "s3_repository",
"config": {
"indices": ["*"]
"retention": {
"expire_after": "30d",
"min_count": 240,
"max_count": 750

any idea ? @Anya_Sabo @sebgl

Looking at the error log:
It looks like the S3 repository plugin is trying to retrieve some metadata from aws metadata server, which does not seem to be accessible from where you're using this plugin?

I'm still wondering whether secure settings are setup correctly. Can you share the content of your Elasticsearch yaml manifest and the content of your secure settings secret (kubectl get secret datawarehouse-aws-credentials -o yaml). Please strip out any secret information from there, including the base64 payload in the secret (we're only interested in the secret keys).

The cluster is running inside GKE.
Our snapshot are stored inside S3.
The es, secure settings and script manifests:
The script was running as kubernetes job and everything was in success

Do you have an idea @sebgl why ILM don't read my secure setings ?

@dg_hivebrite I don't see the secureSettings field set in your elasticsearch.yaml manifest.
It should probably look like:

  - secretName: datawarehouse-aws-credentials

@sebgl yes good catch in my helm template there was an error.
Now I can see the secure settings but the error is not fixed
the elasticsearch manifests + some request on the cluster about slm:

I'm not very familiar with the s3 repository, but by looking around I guess:

ok the last problem was my secret weren't in b64.
Everything works well and the operator upgrade everything :slight_smile:

Happy we got things sorted out @dg_hivebrite :slight_smile:

Just curious where is the base64 requirement for the secret documented? That doesn't add any security, seems like a weird requirement?

@Jamshid that not specific to ECK but rather to Kubernetes secrets, see the docs here:

1 Like