# Elasticsearch run in kubernetes

**URL:** <https://discuss.elastic.co/t/elasticsearch-run-in-kubernetes/160679>\
**Category:** Elasticsearch\
**Created:** [December 13, 2018, 7:25am UTC](https://discuss.elastic.co/t/elasticsearch-run-in-kubernetes/160679 "2018-12-13T07:25:05Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![pytimer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pytimer/32/29331_2.png) [@pytimer](https://discuss.elastic.co/u/pytimer)\
**Post date:** [December 13, 2018, 7:25am UTC](https://discuss.elastic.co/t/elasticsearch-run-in-kubernetes/160679/1 "2018-12-13T07:25:05Z")

</div>

Hi, all

I run Elasticsearch 6.2.2 in Kubernetes and use glusterfs as storage backend.

When i start is works well, everything looks OK. But run after a while, health of that some indices turned Red. I don't know what happen. I am the beginner for the Elasticsearch.

I use local file system as storage backend before, it didn't find this problem.

what is the best solution to share the elastic data among all the kubernetes nodes?

Thanks.

Below is my environments.

```auto
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    version: v6.2.2
  name: elasticsearch-logging
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: elasticsearch-logging
      version: v6.2.2
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
        kubernetes.io/cluster-service: "true"
        version: v6.2.2
    spec:
      containers:
      - env:
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.2
        imagePullPolicy: IfNotPresent
        name: elasticsearch-logging
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 520Mi
        volumeMounts:
        - mountPath: /data
          name: elasticsearch-data
        - mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          name: config
          subPath: elasticsearch.yml
        - mountPath: /usr/share/elasticsearch/config/jvm.options
          name: config
          subPath: jvm.options
      initContainers:
      - command:
        - /sbin/sysctl
        - -w
        - vm.max_map_count=262144
        image: alpine:3.6
        imagePullPolicy: IfNotPresent
        name: sysctl
        securityContext:
          privileged: true
      - command:
        - /bin/bash
        - -c
        - chown -R elasticsearch:elasticsearch /data /usr/share/elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.2
        imagePullPolicy: IfNotPresent
        name: chown
        securityContext:
          runAsUser: 0
        volumeMounts:
        - mountPath: /data
          name: elasticsearch-data
      restartPolicy: Always
      volumes:
      - hostPath:
          path: /opt/cloud/elasticsearch/data
          type: ""
        name: elasticsearch-data
      - configMap:
          defaultMode: 420
          name: elasticsearch-config
        name: config

```

`/opt/cloud/elasticsearch/data` is my glusterfs volume path.

```auto
127.0.0.1:/k8s_glusterfs on /opt/cloud type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

```

> **Elasticsearch logs:**
>
> ```auto
> [2018-12-13T11:33:12,881][WARN][o.e.i.e.Engine] [Bf9TJG6] [kubernetes_cloud_kube-system_2018.12.06][0] failed engine [refresh failed source[schedule]]
> org.apache.lucene.index.CorruptIndexException: Problem reading index from store(MMapDirectory@/data/nodes/0/indices/eOQZp7A6TrqE0U1bXK5NKQ/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@27ba7af7) (resource=store(MMapDirectory@/data/nodes/0/indices/eOQZp7A6TrqE0U1bXK5NKQ/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@27ba7af7))
> at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:140) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:78) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
> ...
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/data/nodes/0/indices/eOQZp7A6TrqE0U1bXK5NKQ/0/index/_2sd.cfe")
> at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
> at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
> at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:108) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
> at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:69) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
> at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
> at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:99) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
> ... 29 more
> Suppressed: org.apache.lucene.index.CorruptIndexException: checksum status indeterminate: remaining=0, please run checkindex for more details (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/data/nodes/0/indices/eOQZp7A6TrqE0U1bXK5NKQ/0/index/_2sd.cfe")))
> ...
> 
> ```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 13, 2018, 7:29am UTC](https://discuss.elastic.co/t/elasticsearch-run-in-kubernetes/160679/2 "2018-12-13T07:29:55Z")

</div>

Based on [this post](https://discuss.elastic.co/t/underlying-file-changed-by-an-external-force/137555/4) there seems to be issues using glusterfs with Elasticsearch, so it is not recommended or supported.

---

<div class="post-metadata">

**Author:** ![pytimer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pytimer/32/29331_2.png) [@pytimer](https://discuss.elastic.co/u/pytimer)\
**Post date:** [December 13, 2018, 8:26am UTC](https://discuss.elastic.co/t/elasticsearch-run-in-kubernetes/160679/3 "2018-12-13T08:26:43Z")

</div>

Thanks for you reply.

Yes, my problem like [https://bugzilla.redhat.com/show\_bug.cgi?id=1379568](https://bugzilla.redhat.com/show_bug.cgi?id=1379568) said.

If i want to use the distributed storage, which storage you recommend?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 13, 2018, 9:08am UTC](https://discuss.elastic.co/t/elasticsearch-run-in-kubernetes/160679/4 "2018-12-13T09:08:45Z")

</div>

We generally [discourage using distributed storage](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/tune-for-indexing-speed.html#_use_faster_hardware) both for performance and stability reasons, so I do not have any recommendations.

---

<div class="post-metadata">

**Author:** ![pytimer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pytimer/32/29331_2.png) [@pytimer](https://discuss.elastic.co/u/pytimer)\
**Post date:** [December 13, 2018, 9:19am UTC](https://discuss.elastic.co/t/elasticsearch-run-in-kubernetes/160679/5 "2018-12-13T09:19:51Z")

</div>

If i not use distributed storage, i should create Elasticsearch cluster in the Kubernetes if i want to keeps data not lost, is it?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 13, 2018, 9:29am UTC](https://discuss.elastic.co/t/elasticsearch-run-in-kubernetes/160679/6 "2018-12-13T09:29:07Z")

</div>

I am not sure what the recommendations and best practices around Kubernetes are, so will leave that for someone else to expand on. Using persistent volumes, possibly mounted as a block store, might be an option though?

---

<div class="post-metadata">

**Author:** ![pytimer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pytimer/32/29331_2.png) [@pytimer](https://discuss.elastic.co/u/pytimer)\
**Post date:** [December 13, 2018, 10:23am UTC](https://discuss.elastic.co/t/elasticsearch-run-in-kubernetes/160679/7 "2018-12-13T10:23:37Z")

</div>

Thanks.

I don't know more about block store, but i will try it.

Thanks again. 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 10, 2019, 10:23am UTC](https://discuss.elastic.co/t/elasticsearch-run-in-kubernetes/160679/8 "2019-01-10T10:23:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.