Problem with S3 Plugin

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

My current EFK logging configuration looks like this:

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: efk
  namespace: logging
spec:
  version: 7.14.0
  secureSettings:
  - secretName: es-s3-backup-aws-creds
  nodeSets:
  - name: logging-cluster
    count: 3
    podTemplate:
      spec:
        initContainers:
        - name: install-plugins
          command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install --batch repository-s3
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 300Gi
        storageClassName: gp2

"es-s3-backup-aws-creds" is stored in logging namespace as a secret with my aws access + secrets keys

but I am getting this error when deploying this configuration using the ECK operator

java.lang.IllegalArgumentException: unknown secure setting [s3.client.secondary.secret_key] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533)
	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478)
	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449)
	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420)
	at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138)
	at org.elasticsearch.node.Node.<init>(Node.java:445)
	at org.elasticsearch.node.Node.<init>(Node.java:281)
2

I am thinking the issue might be that you have not yet created a repository configuration in Elasticsearch that references your secondary client. So when the node boots up it sees these additional settings under the secondary key in the keystore and cannot place them and that's why it bails out. Try creating a snapshot repository via the API that references the secondary client.

3

Thank you @pebrc - i will try that out.

4

@pebrc -

I am seeing this

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[my_s3_repository] repository type [s3] does not exist"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[my_s3_repository] repository type [s3] does not exist"
  },
  "status" : 500
}

here is my CRD yaml to deploy the ECK pods

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: efk
  namespace: logging
spec:
  version: 7.14.0
  nodeSets:
  - name: logging-cluster
    count: 1
    podTemplate:
      spec:
        initContainers:
        - name: install-plugins
          command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install --batch repository-s3
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 300Gi
        storageClassName: gp2

for some reason its still not installing the s3 plugin on the ECK pods

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.