Setup snapshot for s3 plugin

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi,

I'm trying to setup the s3 plugin but I have an issue.
I don't how to format the secret for aws credentials for the operator.
Actually I tried that (with terraform):
resource "kubernetes_secret" "aws_credentials_datawarehouse" {
type = "kubernetes.io/generic"

  metadata {
    name = "datawarehouse-aws-credentials"
  }

  data = {
    "s3.client.default.access_key" = "....."
    "s3.client.default.secret_key" = "....."
  }
}

The operator log an error:
E1212 13:40:26.641312 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:94: Failed to list *v1beta1.Elasticsearch: v1beta1.ElasticsearchList.Items: []v1beta1.Elasticsearch: v1beta1.Elasticsearch.Spec: v1beta1.ElasticsearchSpec.SecureSettings: []v1beta1.SecretSource: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|ttings":["datawareho|..., bigger context ...|rageClassName":"standard"}}]}],"secureSettings":["datawarehouse-aws-credentials"],"updateStrategy":{|...

I tried to check in the github repository but I'm not sure of which format is expected inside the secret

2

The error looks more like a bug in the Elasticsearch resource itself.
Can you share your elasticsearch yaml manifest which references the secure settings secrets?

3

That error looks very similar to the one we sometimes see v1beta1 resources using the v1alpha1 format:
https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-upgrading-eck.html#k8s-convert-manifests

4

Thank you @Anya_Sabo , I forgot secureSettings is a list of map, in my case it was a list of string (the secret name).

5

Ok so the elasticsearch cluster take the secure settings but it seems it don't read or don't understand the secure settings.
error: https://gist.github.com/Dudesons/48e30a2b909c3330ec84a2e4c8791089
The payload to create the snapshot:
{
"schedule": "0 0 * * * ?",
"name": "<hourly-snap-{now/d{yyyy.MM.dd.HH}}>",
"repository": "s3_repository",
"config": {
"indices": ["*"]
},
"retention": {
"expire_after": "30d",
"min_count": 240,
"max_count": 750
}
}

6

any idea ? @Anya_Sabo @sebgl

7

Looking at the error log: http://169.254.169.254/latest/meta-data/iam/security-credentials
It looks like the S3 repository plugin is trying to retrieve some metadata from aws metadata server, which does not seem to be accessible from where you're using this plugin?

I'm still wondering whether secure settings are setup correctly. Can you share the content of your Elasticsearch yaml manifest and the content of your secure settings secret (kubectl get secret datawarehouse-aws-credentials -o yaml). Please strip out any secret information from there, including the base64 payload in the secret (we're only interested in the secret keys).

8

The cluster is running inside GKE.
Our snapshot are stored inside S3.
The es, secure settings and script manifests: https://gist.github.com/Dudesons/e7107413785972391d7d034b6b155c2a
The script was running as kubernetes job and everything was in success

9

Do you have an idea @sebgl why ILM don't read my secure setings ?

10

@dg_hivebrite I don't see the secureSettings field set in your elasticsearch.yaml manifest.
See https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-es-secure-settings.html.
It should probably look like:

spec:
  secureSettings:
  - secretName: datawarehouse-aws-credentials
11

@sebgl yes good catch in my helm template there was an error.
Now I can see the secure settings but the error is not fixed
the elasticsearch manifests + some request on the cluster about slm: https://gist.github.com/Dudesons/1ed9e6cae1c4cd3a81fce566b2f40082

12

I'm not very familiar with the s3 repository, but by looking around I guess:

13

ok the last problem was my secret weren't in b64.
Everything works well and the operator upgrade everything :slight_smile:

14

Happy we got things sorted out @dg_hivebrite :slight_smile:

15

Just curious where is the base64 requirement for the secret documented? That doesn't add any security, seems like a weird requirement?

16

@Jamshid that not specific to ECK but rather to Kubernetes secrets, see the docs here: https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-manually

1 Like