Custom role via Secret not created

[pkaramol (Pantelis Karamolegkos)]

I am trying to create custom roles declaratively in ECK (leveraging the file-based role management), therefore I rely on the usage of Secrets as per the instructions found here.
I am defining the following Secret resource:

▶ k get secret roles-secret -o yaml
apiVersion: v1
data:
  roles.yml: Y2xpY2tfYWRtaW5zOgogIHJ1bl9hczogWyAnY2xpY2tzX3dhdGNoZXJfMScgXQogIGNsdXN0ZXI6IFsgJ21vbml0b3InIF0KICBpbmRpY2VzOgogIC0gbmFtZXM6IFsgJ2V2ZW50cy0qJyBdCiAgICBwcml2aWxlZ2VzOiBbICdyZWFkJyBdCiAgICBmaWVsZF9zZWN1cml0eToKICAgICAgZ3JhbnQ6IFsnY2F0ZWdvcnknLCAnQHRpbWVzdGFtcCcsICdtZXNzYWdlJyBdCiAgICBxdWVyeTogJ3sibWF0Y2giOiB7ImNhdGVnb3J5IjogImNsaWNrIn19Jw==
kind: Secret
metadata:
  annotations:
    helm.fluxcd.io/antecedent: elastic:helmrelease/elastic-stack
  creationTimestamp: "2021-08-05T18:06:06Z"
  name: roles-secret
  namespace: elastic
  resourceVersion: "124906226"
  uid: 6411d36e-ae41-415b-97da-e220b9eb5ede
type: Opaque

The decoded value is:

click_admins:
  run_as: [ 'clicks_watcher_1' ]
  cluster: [ 'monitor' ]
  indices:
  - names: [ 'events-*' ]
    privileges: [ 'read' ]
    field_security:
      grant: ['category', '@timestamp', 'message' ]
    query: '{"match": {"category": "click"}}'

I am also defining the necessary spec.auth.roles section in the Elasticsearch resource that points to the corresponding Secret name:

  spec:
    auth:
      roles:
      - secretName: roles-secret

However the click_admins role is never created.

Any suggestions? (I don't see any error logs in the operator) and other than that the cluster is in green state.

FWIW when I delete the Secret manually for debugging purposes, I do see a the following error in the operator logs.

elastic-operator-0 manager {"log.level":"info","@timestamp":"2021-08-05T19:54:06.139Z","log.logger":"elasticsearch-user","message":"referenced secret not found","service.version":"1.6.0+8326ca8a","service.type":"eck","ecs.version":"1.4.0","namespace":"elastic","es_name":"workablestg9","secret_name":"roles-secret"}

This is the only relevant entry ever appearing in the logs which is kind of weird imho.

Since the secret is (likely) appropriately mounted AND referenced by the spec.auth.roles section of Elasticsearch resource, we shouldn't we be getting some kind of indication why the Role is never created.

edit

I have tried to reproduce this using the simplest example possible.
So in a fresh namespace, I have created the following three resources:
(these manifests are all more or less taken from official documentation examples)

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  namespace: test-elastic
  name: quickstart
spec:
  http:
    tls:
      certificate: {}
      selfSignedCertificate:
        disabled: true
  auth:
    roles:
    - secretName: my-roles-secret
  version: 7.14.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  namespace: test-elastic
  name: quickstart
spec:
  version: 7.14.0
  count: 1
  elasticsearchRef:
    name: quickstart

kind: Secret
apiVersion: v1
metadata:
  namespace: test-elastic
  name: my-roles-secret
stringData:
  roles.yml: |-
    click_admins:
      run_as: [ 'clicks_watcher_1' ]
      cluster: [ 'monitor' ]
      indices:
      - names: [ 'events-*' ]
        privileges: [ 'read' ]
        field_security:
          grant: ['category', '@timestamp', 'message' ]
        query: '{"match": {"category": "click"}}'

I then forward kibana, locally, log in, and here is the end result:

[Thibault_Richard (Thibault)]

Duplicate with Custom role not created when using relevant definition in Secret resource · Issue #4738 · elastic/cloud-on-k8s · GitHub.

This behaviour is to be expected.

you cannot add or manage users in the file realm via the user APIs and you cannot add or manage them in Kibana on the Management / Security / Users page

See File-based user authentication | Elasticsearch Guide [8.11] | Elastic.