[ECK] Secret: Could not resolve placeholder

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi guys,

I've been trying to setup my ECK deployment to use the OpenID with AzureAD, however I can't do the configuration right, probably mistake and lack of knowledge.

Following my cenario:

The error happens in the Pod of Elasticsearch onto container: elastic-init-internal-key-store:

+ keystore_initialized_flag=/usr/share/elasticsearch/config/elastic-internal-init-keystore.ok
+ [[ -f /usr/share/elasticsearch/config/elastic-internal-init-keystore.ok ]]
+ echo 'Initializing keystore.'
Initializing keystore.
+ /usr/share/elasticsearch/bin/elasticsearch-keystore create
Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'oidc_azure_secret'
	at org.elasticsearch.common.settings.PropertyPlaceholder.parseStringValue(PropertyPlaceholder.java:102)
	at org.elasticsearch.common.settings.PropertyPlaceholder.replacePlaceholders(PropertyPlaceholder.java:57)
	at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1314)
	at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:56)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:110)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:94)

Following my elastic-deployment.yaml:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: eck-elasticsearch
  namespace: prod-infra
spec:
  version: 8.14.1
  secureSettings:
  - secretName: oidc-azure-openid
    entries:
      - key: oidc.client_id
      - key: oidc.client_secret
  monitoring:
      metrics:
        elasticsearchRefs:
        - name: eck-elasticsearch
          namespace: prod-infra
      logs:
        elasticsearchRefs:
        - name: eck-elasticsearch
          namespace: prod-infra
  updateStrategy:
    changeBudget:
      maxSurge: 4
      maxUnavailable: 2
##### Master Nodes          
  nodeSets:
  - name: masters
    count: 3
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: gp2    
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
            runAsUser: 0
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
          - name: elasticsearch
            env:
            - name: PRE_STOP_ADDITIONAL_WAIT_SECONDS
              value: "5"
    config:
      node.roles: ["master", "remote_cluster_client"]
      xpack.ml.enabled: true  
      xpack.security.audit.enabled: false
      xpack.security.authc.realms.oidc.oidc1:
        order: 2
        rp.client_id: "${oidc_client_id}"
        rp.client_secret: "${oidc_client_secret}"
        rp.response_type: code
        rp.requested_scopes: ["openid", "email"]
        rp.redirect_uri: "https://xxxxxxxxxx.com/api/security/oidc/callback"
        op.issuer: "https://login.microsoftonline.com/xxxxx-xxxxx-xx-x-x-x-xxxx/v2.0"
        op.authorization_endpoint: "https://login.microsoftonline.com/xxxxx-xxxxx-xx-x-x-x-xxxx/oauth2/authorize"
        op.token_endpoint: "https://login.microsoftonline.com/xxxxx-xxxxx-xx-x-x-x-xxxx/oauth2/token"
        op.jwkset_path: "https://login.microsoftonline.com/xxxxx-xxxxx-xx-x-x-x-xxxx/discovery/v2.0/keys"
        op.userinfo_endpoint: "https://graph.microsoft.com/oidc/userinfo"
        op.endsession_endpoint: "https://login.microsoftonline.com/xxxxx-xxxxx-xx-x-x-x-xxxx/oauth2/v2.0/logout"
        rp.post_logout_redirect_uri: "https://xxxxxxxxxx.com/security/logged_out"
        claims.principal: email
        claims.groups: groups
#### Data Nodes      
  - name: data
    count: 3
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: gp2
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
            runAsUser: 0
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
          - name: elasticsearch
            env:
            - name: PRE_STOP_ADDITIONAL_WAIT_SECONDS
              value: "5"          
    config:
      node.roles: ["data", "ingest", "ml", "transform", "remote_cluster_client"]
      xpack.ml.enabled: true
      xpack.security.audit.enabled: false
      xpack.security.authc.token.enabled: true      
      xpack.security.authc.realms.oidc.oidc1:
        order: 2
        rp.client_id: "${oidc_client_id}"
        rp.client_secret: ${oidc_azure_secret}
        rp.response_type: code
        rp.requested_scopes: ["openid", "email"]
        rp.redirect_uri: "https://xxxx.com/api/security/oidc/callback"
        op.issuer: "https://login.microsoftonline.com/xxxxxxxxxx-x--xxxxxx-xx-xxx/v2.0"
        op.authorization_endpoint: "https://login.microsoftonline.com/xxxxxxxxxx-x--xxxxxx-xx-xxx/oauth2/authorize"
        op.token_endpoint: "https://login.microsoftonline.com/xxxxxxxxxx-x--xxxxxx-xx-xxx/oauth2/token"
        op.jwkset_path: "https://login.microsoftonline.com/xxxxxxxxxx-x--xxxxxx-xx-xxx/discovery/v2.0/keys"
        op.userinfo_endpoint: "https://graph.microsoft.com/oidc/userinfo"
        op.endsession_endpoint: "https://login.microsoftonline.com/xxxxxxxxxx-x--xxxxxx-xx-xxx/oauth2/v2.0/logout"
        rp.post_logout_redirect_uri: "https://xxxx.com/security/logged_out"
        claims.principal: email
        claims.groups: groups

And my kibana-deployment.yaml:

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: eck-kibana
  namespace: prod-infra
spec:
  version: 8.14.1
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
          - name: NODE_OPTIONS
            value: "--max-old-space-size=2048"
        resources:
          requests:
            memory: 1Gi
            cpu: 0.5
          limits:
            memory: 2.5Gi
            cpu: 2
  monitoring:
    metrics:
      elasticsearchRefs:
      - name: eck-elasticsearch
        namespace: prod-infra
    logs:
      elasticsearchRefs:
      - name: eck-elasticsearch
        namespace: prod-infra
  config:
    xpack.security.authc.providers:
      oidc.oidc1:
        order: 0
        realm: oidc1
        description: "Log in with Azure"
      basic.basic1:
        order: 1
    xpack.security.audit.enabled: false
    xpack.security.session.idleTimeout: "15m"
    xpack.fleet.packages:
    - name: apm
      version: latest
    monitoring.ui.ccs.enabled: true
    monitoring.ui.container.elasticsearch.enabled: true
    telemetry.enabled: false
    server.publicBaseUrl: https://xxxx.com      
  count: 1
  elasticsearchRef:
    name: eck-elasticsearch
  http:
    tls:
      selfSignedCertificate:
        disabled: true    

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: eck-kibana-ingress
  namespace: prod-infra
  annotations:
    alb.ingress.kubernetes.io/group.name: ingress-my-ingress
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - host: xxxx.com
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: eck-kibana-kb-http
                port:
                  number: 5601

Is someone know how to fix?

2

Hey :wave:

Good reflex to share your elastic deployment and kibana deployment but here your problem only come from your elastic deployment.

it seems obvious but have you check :

  • if oidc-azure-openid is stocked in your "prod-infra" namespace ?
  • if oidc-azure-openid stock oidc.client_id oidc.client_secret ?

Also you forget to declare "${oidc_client_id}" in InitContainer
I have tried to solve your problem. In case it doesn't work, I hope it provides you with a new perspective.

    podTemplate:
      spec:
        initContainers:
          - name: inject-secrets
            image: busybox
            command: ['sh', '-c', 'echo ${oidc_client_id} > /mnt/secret/oidc_client_id && echo ${oidc_client_secret} > /mnt/secret/oidc_client_secret']
            env:
              - name: oidc_client_id
                valueFrom:
                  secretKeyRef:
                    name: oidc-azure-openid
                    key: oidc.client_id
              - name: oidc_client_secret
                valueFrom:
                  secretKeyRef:
                    name: oidc-azure-openid
                    key: oidc.client_secret
            volumeMounts:
              - name: oidc-secrets
                mountPath: /mnt/secret
          - name: sysctl
            securityContext:
              privileged: true
              runAsUser: 0
            command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
          - name: elasticsearch
            env:
              - name: PRE_STOP_ADDITIONAL_WAIT_SECONDS
                value: "5"
            volumeMounts:
              - name: oidc-secrets
                mountPath: /mnt/secret
        volumes:
          - name: oidc-secrets
            emptyDir: {}