Installing things on ES startup

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

How to make sure something gets installed into ES on startup? Just right now I'm looking at users and roles, but there are probably other things we want to get in through the HTTP API after ES has started up but before clients try to collect to it.

I'm currently getting Logstash to install the one and only template allowed during its startup, but that's obviously not a general solution. I note various discussions about users and roles, but no actual answer?

One approach might be to have each client application install the users and roles that it needs in an init container, but I haven't worked out the security implications of this and it sounds like an awful lot of work.

2

The next 1.1.0 version (available in a few days) gives you a way to define custom roles and users: https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-users-and-roles.html.

More generally, to install things on startup, init containers are a valid option. Another option is to handle that with the tools you use to automate your infrastructure (Ansible, Terraform, Shell scripts, ...) after the ECK operator and your Elastic Stack have been installed.

We are still taking a look into this problematic for specific areas:

3

Thanks - in particular the "next 1.1.0 version" looks interesting.

I'm not getting my head around how init containers are useful, though, because they run before the main container, and surely what I need here is something that runs after the main container is properly up and running but (preferably) before clients can connect to it?

4

You are right, init containers are not suitable to setup things on the main container when it must be alive. I had in mind the usecase of the preparation of configuration files.

That is not perfect and require some work but you could also use a cron job that polls the status of the service you want to configure and will configure things as soon as it detects it is ready.

5

I'm finding this a general problem with trying to use K8s, not restricted to the Elastic stack - we have plenty of other databases and stuff that need initial things poking into them once they're up and running. As I can't be the only one with this problem there must be an answer out there ... but I haven't found it yet ...

6

A possible (but complicated) workaround would be to use a custom readiness gate coupled with a cron job.

  • Add a readiness gate to the pod template:
    podTemplate:
  spec:
    readinessGates:
      - conditionType: "eck.k8s.elastic.co/readyplusplus"
  • Create a cron job that polls the pods until they all come up. Because of the readiness gate, those pods won't be added to the endpoints of the service yet (thereby making the Elasticsearch cluster effectively unavailable for other applications).
  • Once the pods come up, apply your changes through the API by accessing one of the pods directly.
  • Flip the readiness gate on all pods to make the cluster available for traffic via the service.
    for POD in $PODS; do curl -H 'Content-Type: application/strategic-merge-patch+json' -XPATCH "http://localhost:8001/api/v1/namespaces/default/pods/$POD/status" -d '{"status":{"conditions":[{"type":"eck.k8s.elastic.co/readyplusplus","status": "True"}]}}'; done