I'm running EFK Stack on my EKS cluster and i'm mounting elasticsearch volume on EFS so my data won't be lost when the container is restarted, however after few months my elasticsearch pod raised this exception "Disk quota exceeded" i think its related to storage space.
Here is elasticsearch deployment :
apiVersion: apps/v1
kind: Deployment
metadata:
name: elasticsearch
namespace: tools
spec:
replicas: 1
selector:
matchLabels:
app: elasticsearch
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- env:
- name: discovery.type
valueFrom:
configMapKeyRef:
key: discovery.type
name: tools-config
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 20
periodSeconds: 5
successThreshold: 1
tcpSocket:
port: transport
timeoutSeconds: 1
name: elasticsearch
ports:
- containerPort: 9200
name: http
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /_cluster/health
port: http
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
volumeMounts:
- mountPath: /usr/share/elasticsearch/data
name: elasticsearch-volume
initContainers:
- command:
- sh
- -c
- chown -R 1000:1000 /usr/share/elasticsearch/data
image: busybox
imagePullPolicy: Always
name: fix-permissions
securityContext:
privileged: true
volumeMounts:
- mountPath: /usr/share/elasticsearch/data
name: elasticsearch-volume
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: elasticsearch-volume
persistentVolumeClaim:
claimName: elasticsearch-storage-pvc