Deploy Elastic Search with Kubernetes StatefulSet cannot persist the index data

[Bai]

Hi All,

I'm deploying Elastic Search as StatefulSet in Kubernetes, but cannot make the persistent volume working properly. (Reference: https://github.com/pires/kubernetes-elasticsearch-cluster/tree/master/stateful, I don't create dedicated master, so simply master & data are in one).

I'm testing the persistent volume by:

1. create ES and insert some data;
2. delete and re-create the service;
3. check if the index is still there.

The deployment showed all green, and the Elasticsearch is also working fine. But the data is not persistent if I re-create the ES. Could someone shed some light on this? Thanks!

Attached some stats, and let me know if I need to post other logs.

kubectl describe pvc

Name:          storage-elasticsearch-logging-v1-0
Namespace:     kube-system
StorageClass:  standard
Status:        Bound
Volume:        pvc-96453dd3-36c9-11e8-9f27-029b836b1aaa
Labels:        k8s-app=elasticsearch-logging
               kubernetes.io/cluster-service=true
               version=v1
Annotations:   pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
               volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/aws-ebs
Finalizers:    []
Capacity:      2Gi
Access Modes:  RWO
Events:        <none>

kubectl describe pv

Name:            pvc-a1825ec9-36c9-11e8-9f27-029b836b1aaa
Labels:          failure-domain.beta.kubernetes.io/region=us-west-2
                 failure-domain.beta.kubernetes.io/zone=us-west-2a
Annotations:     kubernetes.io/createdby=aws-ebs-dynamic-provisioner
                 pv.kubernetes.io/bound-by-controller=yes
                 pv.kubernetes.io/provisioned-by=kubernetes.io/aws-ebs
StorageClass:    standard
Status:          Bound
Claim:           kube-system/storage-elasticsearch-logging-v1-1
Reclaim Policy:  Retain
Access Modes:    RWO
Capacity:        2Gi
Message:
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   aws://us-west-2a/vol-01aa1c6a2ad2eae3a
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>

kubectl describe sc standard

Name:            standard
IsDefaultClass:  No
Annotations:     <none>
Provisioner:     kubernetes.io/aws-ebs
Parameters:      type=gp2
ReclaimPolicy:   Retain
Events:          <none>

[DavidTurner]

Elasticsearch keeps its data in the configured data directories. It seems likely that path.data isn't pointing to the persistent disk, so it's writing its data somewhere else that is lost when you shut the container down.

[Bai]

Thanks David! Indeed that was the case, as we also figured the ES data is stored in "/usr/share/elasticsearch" while we were mounting "/data" folder.