How to specify path.data on startup of 6.3.1

[Mike_H (Mike Hardy)]

Hi, I want to persist data persist when our Kubernetes StatefulSets are destroyed and recreated. I have one master node and one data node.

I think, like this problem, the issue is setting path.data, but I'm not completely sure how to set the variable. I've tried a few different methods like manually setting the value in elasticsearch.yml but I've had zero success. Ideally, I would love to set the path on startup. Any advice is much appreciated!

Here's the StatefulSet yaml for our data node:

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: elastic-data
  labels:
    app: elastic-data
    area: devs
    role: nosql
    version: "6.3.1"
    environment: elastic
spec:
  serviceName: elastic-data
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: elastic-data
        area: devs
        role: nosql
        version: "6.3.1"
        environment: elastic
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
          {
          "name": "sysctl",
            "image": "busybox",
            "imagePullPolicy": "IfNotPresent",
            "command": ["sysctl", "-w", "vm.max_map_count=262144"],
            "securityContext": {
              "privileged": true
            }
          }
        ]'
    spec:
      terminationGracePeriodSeconds: 10
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
      containers:
      - name: elastic-data
        image: docker.elastic.co/elasticsearch/elasticsearch:6.3.1
        env:
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        command: ["/bin/bash", "-c", "~/bin/elasticsearch-plugin remove x-pack; sed -i.bak -e /xpack.license.self_generated.type/d config/elasticsearch.yml; elasticsearch"]
        args:
        - -Ecluster.name=elastic-devs
        - -Enode.name=${HOSTNAME}
        - -Ediscovery.zen.ping.unicast.hosts=elastic-master.default.svc.cluster.local
        - -Enode.master=false
        - -Enode.data=true
        - -Enode.ingest=false
        - -Enetwork.host=0.0.0.0
        - -Expack.security.enabled=false
        resources:
          requests:
            memory: "512Mi"
          limits:
            memory: "1024Mi"
        ports:
        - containerPort: 9300
          name: transport
        - containerPort: 9200
          name: http
        volumeMounts:
        - name: data-volume
          mountPath: /usr/share/elasticsearch/data
        readinessProbe:
          tcpSocket:
            port: 9300
          initialDelaySeconds: 30
          periodSeconds: 30
          timeoutSeconds: 3
        livenessProbe:
          tcpSocket:
            port: 9300
          initialDelaySeconds: 30
          periodSeconds: 30
          timeoutSeconds: 3
      volumes:
      - name: data-volume
        persistentVolumeClaim:
          claimName: pv-claim-es

[Bai]

@Mike_H: You can mount your own version of elasticsearch.yml like:

    volumeMounts:
        - name: storage
          mountPath: /usr/share/elasticsearch/data
        - name: config-volume
          mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          subPath: elasticsearch.yml

  volumes:
  - name: config-volume
    configMap:
      name: elasticsearch-config

In the configMap you created with your own version of elasticsearch.yml, you can config the data path below so the volume you mount will match the actual data folder ES is writing to:

apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-config
data:
  elasticsearch.yml: |
    path:
      data: ${DATA_PATH:/usr/share/elasticsearch/data}

This is how I resolved the issue before, hope it can help.

[Mike_H (Mike Hardy)]

@Bai Thanks for the help! I've tried something like this in the past and I just tried it again with your elasticsearch.yml file but I'm getting this setting on startup. Did you ever come across this?

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [data.elasticsearch.yml] please check that any required plugins are installed, or check the breaking changes documentation for removed settings at org.elasticsearch.bootstrap.Elasticsearch.init

[Bai]

No, never saw this.

Here is the full configMap I'm using for your reference:

apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-config
  namespace: backend
data:
  elasticsearch.yml: |
    cluster:
      name: ${CLUSTER_NAME:elasticsearch-default}

    node:
      master: ${NODE_MASTER:true}
      data: ${NODE_DATA:true}
      name: ${NODE_NAME}
      ingest: ${NODE_INGEST:true}
      max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES:1}

    processors: ${PROCESSORS:1}

    network.host: ${NETWORK_HOST:_site_}

    path:
      data: ${DATA_PATH:"/usr/share/elasticsearch/data"}
      repo: ${REPO_LOCATIONS:[]}

    bootstrap:
      memory_lock: ${MEMORY_LOCK:false}

    http:
      enabled: ${HTTP_ENABLE:true}
      compression: true
      cors:
        enabled: true
        allow-origin: "*"

    discovery:
      zen:
        ping.unicast.hosts: ${DISCOVERY_SERVICE:elasticsearch-discovery}
        minimum_master_nodes: ${NUMBER_OF_MASTERS:1}

    xpack:
      license.self_generated.type: basic

[Mike_H (Mike Hardy)]

Thank Bai! I had an issue in my formatting that tossed the error. It looks like it's solved now - thanks again