Hi, all
I run Elasticsearch 6.2.2 in Kubernetes and use glusterfs as storage backend.
When i start is works well, everything looks OK. But run after a while, health of that some indices turned Red. I don't know what happen. I am the beginner for the Elasticsearch.
I use local file system as storage backend before, it didn't find this problem.
what is the best solution to share the elastic data among all the kubernetes nodes?
Thanks.
Below is my environments.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
k8s-app: elasticsearch-logging
kubernetes.io/cluster-service: "true"
version: v6.2.2
name: elasticsearch-logging
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
k8s-app: elasticsearch-logging
version: v6.2.2
template:
metadata:
labels:
k8s-app: elasticsearch-logging
kubernetes.io/cluster-service: "true"
version: v6.2.2
spec:
containers:
- env:
- name: ES_JAVA_OPTS
value: -Xms512m -Xmx512m
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.2
imagePullPolicy: IfNotPresent
name: elasticsearch-logging
ports:
- containerPort: 9200
name: db
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
resources:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: 100m
memory: 520Mi
volumeMounts:
- mountPath: /data
name: elasticsearch-data
- mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
name: config
subPath: elasticsearch.yml
- mountPath: /usr/share/elasticsearch/config/jvm.options
name: config
subPath: jvm.options
initContainers:
- command:
- /sbin/sysctl
- -w
- vm.max_map_count=262144
image: alpine:3.6
imagePullPolicy: IfNotPresent
name: sysctl
securityContext:
privileged: true
- command:
- /bin/bash
- -c
- chown -R elasticsearch:elasticsearch /data /usr/share/elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.2
imagePullPolicy: IfNotPresent
name: chown
securityContext:
runAsUser: 0
volumeMounts:
- mountPath: /data
name: elasticsearch-data
restartPolicy: Always
volumes:
- hostPath:
path: /opt/cloud/elasticsearch/data
type: ""
name: elasticsearch-data
- configMap:
defaultMode: 420
name: elasticsearch-config
name: config
/opt/cloud/elasticsearch/data is my glusterfs volume path.
127.0.0.1:/k8s_glusterfs on /opt/cloud type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
Elasticsearch logs:
[2018-12-13T11:33:12,881][WARN ][o.e.i.e.Engine ] [Bf9TJG6] [kubernetes_cloud_kube-system_2018.12.06][0] failed engine [refresh failed source[schedule]]
org.apache.lucene.index.CorruptIndexException: Problem reading index from store(MMapDirectory@/data/nodes/0/indices/eOQZp7A6TrqE0U1bXK5NKQ/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@27ba7af7) (resource=store(MMapDirectory@/data/nodes/0/indices/eOQZp7A6TrqE0U1bXK5NKQ/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@27ba7af7))
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:140) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:78) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
...
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/data/nodes/0/indices/eOQZp7A6TrqE0U1bXK5NKQ/0/index/_2sd.cfe")
at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:108) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:69) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:99) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
... 29 more
Suppressed: org.apache.lucene.index.CorruptIndexException: checksum status indeterminate: remaining=0, please run checkindex for more details (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/data/nodes/0/indices/eOQZp7A6TrqE0U1bXK5NKQ/0/index/_2sd.cfe")))
...
