How to make elasticsearch.keystore persistent?

Elastic Stack Elasticsearch
1

Hi, I am running elasticsearch cluster on Kubernetes cluster. The passwords I have generated for build-in-users like kibana_system and elastic are changed after restart of the elasticsearch node. I am trying to find a way to make them persistent, so after restart they will remain the same.
I have tried to add them to the keystore and mount it to be persistent but it doesn't work that way.

What I have tried:

  1. ./bin/elasticsearch-keystore add kibana_system
    ./bin/elasticsearch-keystore add elastic
  2. I copied /usr/search/elasticsearch/config/elasticsearch.keystore locally
  3. I created configmap
    kubectl create configmap elasticsearch-keystore —from-file=elasticsearch.keystore
  4. In the deployment in volumeMounts I added
    • mountPath: /usr/search/elasticsearch/config/elasticsearch.keystore
      name: keystore
      subPath: elasticsearch.keystore
  5. In the deployment in volumes I added

Is there any other way to do this?

2

Welcome to our community! :smiley:

Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

Are your container volumes also persistent? Are you using ECK?

3

Hi, I am using Vanilla Kubernetes.

This is the configmap of the client node:

apiVersion: v1
data:
  elasticsearch.yml: |-
    cluster.name: ${CLUSTER_NAME}
    node.name: ${NODE_NAME}
    discovery.seed_hosts: ${NODE_LIST}
    cluster.initial_master_nodes: ${MASTER_NODES}
    network.host: 0.0.0.0
    node.roles: []
    xpack.monitoring.collection.enabled: true
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.enabled: false
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.authc.realms.native.native1.order: 0
    xpack.security.authc.realms.native.native1.enabled: true
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"elasticsearch.yml":"cluster.name: ${CLUSTER_NAME}\nnode.name: ${NODE_NAME}\ndiscovery.seed_hosts: ${NODE_LIST}\ncluster.initial_master_nodes: ${MASTER_NODES}\nnetwork.host: 0.0.0.0\nnode.roles: []\nxpack.security.enabled: true\nxpack.security.transport.ssl.enabled: true\nxpack.security.transport.ssl.verification_mode: certificate\nxpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12\nxpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12\nxpack.monitoring.collection.enabled: true"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"elasticsearch","role":"client"},"name":"elasticsearch-client-config","namespace":"logging-kubernetes"}}
  creationTimestamp: "2023-04-11T15:28:28Z"
  labels:
    app: elasticsearch
    role: client
  name: elasticsearch-client-config
  namespace: logging-kubernetes
  resourceVersion: "198917760"
  uid: 08e62e1c-852a-4420-aac8-484f1770e717

This is the deployment of the client node:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "49"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"elasticsearch","role":"client"},"name":"elasticsearch-client","namespace":"logging-kubernetes"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"elasticsearch","role":"client"}},"template":{"metadata":{"labels":{"app":"elasticsearch","role":"client"}},"spec":{"containers":[{"env":[{"name":"CLUSTER_NAME","value":"elasticsearch"},{"name":"NODE_NAME","value":"elasticsearch-client"},{"name":"NODE_LIST","value":"elasticsearch-master,elasticsearch-data,elasticsearch-client"},{"name":"MASTER_NODES","value":"elasticsearch-master"},{"name":"ES_JAVA_OPTS","value":"-Xms256m -Xmx256m"}],"image":"docker.elastic.co/elasticsearch/elasticsearch:8.7.0","name":"elasticsearch-client","ports":[{"containerPort":9200,"name":"client"},{"containerPort":9300,"name":"transport"}],"volumeMounts":[{"mountPath":"/usr/share/elasticsearch/config/elasticsearch.yml","name":"config","readOnly":true,"subPath":"elasticsearch.yml"},{"mountPath":"/data","name":"storage"},{"mountPath":"/usr/share/elasticsearch/config/certs","name":"elastic-certificates"}]}],"initContainers":[{"command":["sysctl","-w","vm.max_map_count=262144"],"image":"busybox","name":"increase-vm-max-map","securityContext":{"privileged":true}}],"volumes":[{"configMap":{"name":"elasticsearch-client-config"},"name":"config"},{"name":"elastic-certificates","secret":{"secretName":"elastic-certificate-pem"}},{"emptyDir":{"medium":""},"name":"storage"}]}}}}
  creationTimestamp: "2023-04-18T10:29:20Z"
  generation: 49
  labels:
    app: elasticsearch
    role: client
  name: elasticsearch-client
  namespace: logging-kubernetes
  resourceVersion: "199314144"
  uid: f11473aa-f88a-4987-a4f1-633618f36706
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: elasticsearch
      role: client
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2023-05-05T10:10:10+02:00"
      creationTimestamp: null
      labels:
        app: elasticsearch
        role: client
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - elasticsearch
                  - elasticsearch-data
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - env:
        - name: CLUSTER_NAME
          value: elasticsearch
        - name: NODE_NAME
          value: elasticsearch-client
        - name: NODE_LIST
          value: elasticsearch-master,elasticsearch-data,elasticsearch-client
        - name: MASTER_NODES
          value: elasticsearch-master
        - name: ES_JAVA_OPTS
          value: -Xms256m -Xmx256m
        image: docker.elastic.co/elasticsearch/elasticsearch:8.7.0
        imagePullPolicy: IfNotPresent
        name: elasticsearch-client
        ports:
        - containerPort: 9200
          name: client
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          name: config
          readOnly: true
          subPath: elasticsearch.yml
        - mountPath: /data
          name: storage
        - mountPath: /usr/share/elasticsearch/config/certs
          name: elastic-certificates
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        image: busybox
        imagePullPolicy: Always
        name: increase-vm-max-map
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: elasticsearch-client-config
        name: config
      - name: elastic-certificates
        secret:
          defaultMode: 420
          secretName: elastic-certificate-pem
      - emptyDir: {}
        name: storage
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2023-05-04T15:26:16Z"
    lastUpdateTime: "2023-05-04T15:26:16Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2023-04-18T10:29:20Z"
    lastUpdateTime: "2023-05-05T08:10:17Z"
    message: ReplicaSet "elasticsearch-client-696f674558" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 49
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

I want to deploy EFK stack on Kubernetes, and as I found I need kibana_system user so Kibana and Elaticsearch can communicate. But if the pod restarts the password is changed and I need to generate new password, and because of this I want to find a way to make this password persistent.

4

I don't know kubernetes all that well tbh, but this sounds like you are not using persistent volumes for your pods, so it's losing the datae.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.