Elasticsearch Helm chart clarification

[Benoit_Martin]

Hello,

I need help to understand a little better the Elasticsearch helm chart. I used it to deploy a simple cluster of 3 nodes. I specified wanting a persistent volume of 30 gig per node. Everything seems well deployed and I use the cluster for about 1 week. On the other hand I just noticed that each POD contains more than 1 TB and that the /usr/share/elasticsearch/data directory is 492 gig.

Thanks for your help.

[Benoit_Martin]

Here is a copy of the values.yml file and the command I used to deploy it all.

helm template --name elasticsearch ./elasticsearch --set imageTag=7.3.2 \ --set volumeClaimTemplate.storageClassName=ourstorageclass \ --namespace=ournamespace | kubectl -n ournamespace apply -f -

I prefer to use template instead of the Tiller. I don't have access to the whole cluster right.

---
clusterName: "elasticsearch"
nodeGroup: "master"

# The service that non master groups will try to connect to when joining the cluster
# This should be set to clusterName + "-" + nodeGroup for your master group
masterService: ""

# Elasticsearch roles that will be applied to this nodeGroup
# These will be set as environment variables. E.g. node.master=true
roles:
  master: "true"
  ingest: "true"
  data: "true"

replicas: 3
minimumMasterNodes: 2

esMajorVersion: ""

# Allows you to add any config files in /usr/share/elasticsearch/config/
# such as elasticsearch.yml and log4j2.properties
esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.enabled: false


# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.3.2"
imagePullPolicy: "IfNotPresent"

podAnnotations: {}
  # iam.amazonaws.com/role: es-cluster

# additionals labels
labels: {}

esJavaOpts: "-Xmx8g -Xms8g"

resources: {}

initResources: {}

sidecarResources: {}

networkHost: "0.0.0.0"

volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 30Gi

rbac:
  create: false
  serviceAccountName: ""

podSecurityPolicy:
  create: false
  name: ""
  spec:
    privileged: true
    fsGroup:
      rule: RunAsAny
    runAsUser:
      rule: RunAsAny
    seLinux:
      rule: RunAsAny
    supplementalGroups:
      rule: RunAsAny
    volumes:
      - secret
      - configMap
      - persistentVolumeClaim

persistence:
  enabled: true
  annotations: {}

extraVolumes: ""
  # - name: extras
  #   emptyDir: {}

extraVolumeMounts: ""

extraInitContainers: ""

# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""

# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"

# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "hard"

# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}

# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"

protocol: http
httpPort: 9200
transportPort: 9300

service:
  type: ClusterIP
  nodePort:
  annotations: {}
  httpPortName: http
  transportPortName: transport

updateStrategy: RollingUpdate

# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1

podSecurityContext:
  fsGroup: 1000
  runAsUser: 1000

# The following value is deprecated,
# please use the above podSecurityContext.fsGroup instead
fsGroup: ""

securityContext:
  capabilities:
    drop:
    - ALL
  # readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000

# How long to wait for elasticsearch to stop gracefully
terminationGracePeriod: 120

sysctlVmMaxMapCount: 262144

readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5

# https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params wait_for_status
clusterHealthCheckParams: "wait_for_status=green&timeout=1s"

## Use an alternate scheduler.
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
schedulerName: ""

imagePullSecrets: []
nodeSelector: {}
tolerations: []

# Enabling this will publically expose your Elasticsearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
  enabled: true
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - myhostname.domain.name
  tls:
    - secretName: elasticsearch-ingress-cert
      hosts:
        - myhostname.domain.name

nameOverride: ""
fullnameOverride: ""

# https://github.com/elastic/helm-charts/issues/63
masterTerminationFix: false

lifecycle: {}


sysctlInitContainer:
  enabled: true

keystore: []

[Benoit_Martin]

I used a NFS storage class so I think the PVC see the total mount space. I will change to an object storage. By cons I do not understand the overlay of 1TB yet.

[elasticsearch@elasticsearch-master-0 ~]$ df -h
Filesystem                                                     Size  Used Avail Use% Mounted on
overlay                                                        1.1T  207G  837G  20% /
tmpfs                                                           64M     0   64M   0% /dev
tmpfs                                                          126G     0  126G   0% /sys/fs/cgroup
/dev/sdc2                                                      1.1T  207G  837G  20% /etc/hosts
shm                                                             64M     0   64M   0% /dev/shm
10.43.95.255:/export/pvc-84d105c3-45ea-4467-8d49-e5679386d901  492G   20G  473G   4% /usr/share/elasticsearch/data
tmpfs                                                          126G   12K  126G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                          126G  4.0K  126G   1% /usr/share/elasticsearch/config/certs
tmpfs                                                          126G     0  126G   0% /proc/acpi
tmpfs                                                          126G     0  126G   0% /proc/scsi
tmpfs                                                          126G     0  126G   0% /sys/firmware

Thank you!