Elasticsearch build-in user changed password

Elastic Stack Elasticsearch
1

Hi all

Our Elasticsearch environment is deployed on our kubernetes platform using the official helm chart.
We only have 2 elasticsearch pods. We are using xpack.security.enabled: true and setup the password for elastic build-in account by creating a kubernetes secret and using extraEnvs for the chart.

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

We have a requirement to change the password every 3 months for users that is able to login on kibana this includes the elastic built-in account. Upon trying to change the elastic built-in account am getting this error and my cluster stop.

{"type": "server", "timestamp": "2021-06-17T08:47:32,208Z", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elastic", "node.name": "elastic-master-0", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "cluster.uuid": "02_vnO8NQs2sXEawOdE6mQ", "node.id": "-wQ_uf-VR52tzq_UyXR-Og"  }
{"type": "server", "timestamp": "2021-06-17T08:47:42,195Z", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elastic", "node.name": "elastic-master-0", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "cluster.uuid": "02_vnO8NQs2sXEawOdE6mQ", "node.id": "-wQ_uf-VR52tzq_UyXR-Og"  }
{"type": "server", "timestamp": "2021-06-17T08:47:44,287Z", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elastic", "node.name": "elastic-master-0", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "cluster.uuid": "02_vnO8NQs2sXEawOdE6mQ", "node.id": "-wQ_uf-VR52tzq_UyXR-Og"  }

Does this mean in order to change the password for elastic built-in account I have to redeploy the cluster again? Creating a new kubernetes secrets and using that for the deployment of elastic.

Or is there a way for the elastic built in account not able to log-in to kibana but still able to talk to other nodes in elastic?

2

How are you resetting the password every 3 months at the moment?

3

The problem here is that the readiness probe uses $ELASTIC_USERNAME and $ELASTIC_PASSWORD, so if you change the password for $ELASTIC_USERNAME, the readiness probe fails.

I think what you want to do here is stop using elastic for your $ELASTIC_USERNAME. That's a good change to make anyway, because elastic is a superuser, and it's preferable not to run probes as a superuser.

Create a new user with the monitor cluster privilege so it can run the readiness probe. It will not be able to login to Kibana, so it will not fall under your password policy.

Then change $ELASTIC_USER to use that new username instead, and you should be safe to change the elastic password after that.
I suspect (but I'm not an expert since I don't work on the Helm charts) that the initial password for your elastic user (on a brand new cluster) will default to $ELASTIC_PASSWORD.

You could open issue on the Helm GitHub repo and ask about support for a specific PROBE_USERNAME and PROBE_PASSWORD so this is a bit simpler. It seems like a reasonable feature (but it's not something I work on, so there may be reasons why it's not as useful as it sounds).

1 Like
4

Currently done manually. When we have the license we are planning to use our existing ldap for the users. Our ldap configuration changes the users password every 3 months.

5

Are there any of these built-in users have the privillage monitoring cluster privilege?

  • kibana_system
  • logstash_system
  • logstash_system
  • apm_system
  • remote_monitoring_user

The reason why am asking is am not sure what is the sequence how can I create a new user, using the helm chart. Since the the user with the monitor cluster privilege wont be able to log-in on kibana there is no need to change the password as part of our policy.

6

remote_monitoring_user has monitor access, but I'm not sure how you will set its initial password to something you can use in the readiness probe.

Instead, I think you will need to create a new user in the file realm as part of the Helm chart.

I am not an expert on k8s or helm, but I think you should be able to use the postStart lifecycle hook to create a user and assign it a role that has access to perform monitoring - e.g. remote_monitoring_collector or a custom role defined in roles.yml.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.