Hi I try to use a nfs server to run some backups:
I am configuring elastic.yml in next way:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
namespace: rentallo-elastic
spec:
version: 8.4.2
nodeSets:
- name: default
count: 1
podTemplate:
spec:
securityContext:
runAsGroup: 0
runAsUser: 1000
fsGroup: 0
config:
node.store.allow_mmap: false
path.repo: ["/var/nfs/"]
Also, I have created a directory (a mount this) in the node where elastic is located).
I have checked user id, user group in the node running next command:
id -u elasticsearch -> 1000
id -g elasticsearch -> 0
I have changed the permission for the forder /var/nfs
useradd elasticsearch
sudo chown elasticsearch var/nfs
But anyway I am getting the next error when I restart elastic pod.
{"@timestamp":"2022-12-07T19:55:24.631Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart","error.type":"java.lang.IllegalStateException","error.message":"Unable to access 'path.repo' (/var/nfs)","error.stack_trace":"java.lang.IllegalStateException: Unable to access 'path.repo' (/var/nfs)\n\tat org.elasticsearch.server@8.4.2/org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:66)\n\tat org.elasticsearch.server@8.4.2/org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:250)\n\tat org.elasticsearch.server@8.4.2/org.elasticsearch.bootstrap.Security.createPermissions(Security.java:176)\n\tat org.elasticsearch.server@8.4.2/org.elasticsearch.bootstrap.Security.configure(Security.java:123)\n\tat org.elasticsearch.server@8.4.2/org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:186)\n\tat org.elasticsearch.server@8.4.2/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66)\nCaused by: java.nio.file.AccessDeniedException: /var/nfs\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)\n\tat java.base/java.nio.file.Files.createDirectory(Files.java:700)\n\tat java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)\n\tat java.base/java.nio.file.Files.createDirectories(Files.java:793)\n\tat org.elasticsearch.server@8.4.2/org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:326)\n\tat org.elasticsearch.server@8.4.2/org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:64)\n\t... 5 more\n"}
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/quickstart.log
elastic-internal-init-filesystem:
Container ID: containerd://a51983187c71753082aa2739601b7c49fc2c98e60ef9392255b6b049242014c4
Image: docker.elastic.co/elasticsearch/elasticsearch:8.4.2
Image ID: docker.elastic.co/elasticsearch/elasticsearch@sha256:e051b3a697c3846ac56f84a6adc1e66a7f4f2b03002e69e859d0eb9d7bf9c028
Port: <none>
Host Port: <none>
Command:
bash
-c
/mnt/elastic-internal/scripts/prepare-fs.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 07 Dec 2022 19:54:24 +0000
Finished: Wed, 07 Dec 2022 19:54:25 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 100m
memory: 50Mi
Environment:
POD_IP: (v1:status.podIP)
POD_NAME: quickstart-es-default-0 (v1:metadata.name)
NODE_NAME: (v1:spec.nodeName)
NAMESPACE: rentallo-elastic (v1:metadata.namespace)
PROBE_PASSWORD_PATH: /mnt/elastic-internal/probe-user/elastic-internal-probe
PROBE_USERNAME: elastic-internal-probe
READINESS_PROBE_PROTOCOL: https
HEADLESS_SERVICE_NAME: quickstart-es-default
NSS_SDB_USE_CACHE: no
Mounts:
/mnt/elastic-internal/downward-api from downward-api (ro)
/mnt/elastic-internal/elasticsearch-bin-local from elastic-internal-elasticsearch-bin-local (rw)
/mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
/mnt/elastic-internal/elasticsearch-config-local from elastic-internal-elasticsearch-config-local (rw)
/mnt/elastic-internal/elasticsearch-plugins-local from elastic-internal-elasticsearch-plugins-local (rw)
/mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
/mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
/mnt/elastic-internal/transport-certificates from elastic-internal-transport-certificates (ro)
/mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
/mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
/usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
/usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
/usr/share/elasticsearch/data from elasticsearch-data (rw)
/usr/share/elasticsearch/logs from elasticsearch-logs (rw)
elastic-internal-suspend:
Container ID: containerd://6f9c012e8a153144c07a11b2421b842cdefa0850fe2419ace0e2a00abb9ff9dd
Image: docker.elastic.co/elasticsearch/elasticsearch:8.4.2
Image ID: docker.elastic.co/elasticsearch/elasticsearch@sha256:e051b3a697c3846ac56f84a6adc1e66a7f4f2b03002e69e859d0eb9d7bf9c028
Port: <none>
Host Port: <none>
Command:
bash
-c
/mnt/elastic-internal/scripts/suspend.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 07 Dec 2022 19:54:27 +0000
Finished: Wed, 07 Dec 2022 19:54:27 +0000
Ready: True
Restart Count: 0
Limits:
memory: 2Gi
Requests:
memory: 2Gi
Environment:
POD_IP: (v1:status.podIP)
POD_NAME: quickstart-es-default-0 (v1:metadata.name)
NODE_NAME: (v1:spec.nodeName)
NAMESPACE: rentallo-elastic (v1:metadata.namespace)
PROBE_PASSWORD_PATH: /mnt/elastic-internal/probe-user/elastic-internal-probe
PROBE_USERNAME: elastic-internal-probe
READINESS_PROBE_PROTOCOL: https
HEADLESS_SERVICE_NAME: quickstart-es-default
NSS_SDB_USE_CACHE: no
Mounts:
/mnt/elastic-internal/downward-api from downward-api (ro)
/mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
/mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
/mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
/mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
/mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
/usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
/usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
/usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
/usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
/usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
/usr/share/elasticsearch/data from elasticsearch-data (rw)
/usr/share/elasticsearch/logs from elasticsearch-logs (rw)
/usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Containers:
elasticsearch:
Container ID: containerd://ba1a6b3a2ed9b8b794e41bff44b5047f73563c5ee12ac817899b82aed9f5f176
Image: docker.elastic.co/elasticsearch/elasticsearch:8.4.2
Image ID: docker.elastic.co/elasticsearch/elasticsearch@sha256:e051b3a697c3846ac56f84a6adc1e66a7f4f2b03002e69e859d0eb9d7bf9c028
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 07 Dec 2022 20:00:42 +0000
Finished: Wed, 07 Dec 2022 20:00:47 +0000
Ready: False
Restart Count: 6
Limits:
memory: 2Gi
Requests:
memory: 2Gi
Readiness: exec [bash -c /mnt/elastic-internal/scripts/readiness-probe-script.sh] delay=10s timeout=5s period=5s #success=1 #failure=3
Environment:
POD_IP: (v1:status.podIP)
POD_NAME: quickstart-es-default-0 (v1:metadata.name)
NODE_NAME: (v1:spec.nodeName)
NAMESPACE: rentallo-elastic (v1:metadata.namespace)
PROBE_PASSWORD_PATH: /mnt/elastic-internal/probe-user/elastic-internal-probe
PROBE_USERNAME: elastic-internal-probe
READINESS_PROBE_PROTOCOL: https
HEADLESS_SERVICE_NAME: quickstart-es-default
NSS_SDB_USE_CACHE: no
Mounts:
/mnt/elastic-internal/downward-api from downward-api (ro)
/mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
/mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
/mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
/mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
/mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
/usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
/usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
/usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
/usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
/usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
/usr/share/elasticsearch/data from elasticsearch-data (rw)
/usr/share/elasticsearch/logs from elasticsearch-logs (rw)
/usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
elasticsearch-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: elasticsearch-data-quickstart-es-default-0
ReadOnly: false
downward-api:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels -> labels
elastic-internal-elasticsearch-bin-local:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
elastic-internal-elasticsearch-config:
Type: Secret (a volume populated by a Secret)
SecretName: quickstart-es-default-es-config
Optional: false
elastic-internal-elasticsearch-config-local:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
elastic-internal-elasticsearch-plugins-local:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
elastic-internal-http-certificates:
Type: Secret (a volume populated by a Secret)
SecretName: quickstart-es-http-certs-internal
Optional: false
elastic-internal-probe-user:
Type: Secret (a volume populated by a Secret)
SecretName: quickstart-es-internal-users
Optional: false
elastic-internal-remote-certificate-authorities:
Type: Secret (a volume populated by a Secret)
SecretName: quickstart-es-remote-ca
Optional: false
elastic-internal-scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: quickstart-es-scripts
Optional: false
elastic-internal-transport-certificates:
Type: Secret (a volume populated by a Secret)
SecretName: quickstart-es-default-es-transport-certs
Optional: false
elastic-internal-unicast-hosts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: quickstart-es-unicast-hosts
Optional: false
elastic-internal-xpack-file-realm:
Type: Secret (a volume populated by a Secret)
SecretName: quickstart-es-xpack-file-realm
Optional: false
elasticsearch-logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned rentallo-elastic/quickstart-es-default-0 to k3s-worker-n0-zao
Normal Pulled 9m55s kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.4.2" already present on machine
Normal Created 9m55s kubelet Created container elastic-internal-init-filesystem
Normal Started 9m55s kubelet Started container elastic-internal-init-filesystem
Normal Pulled 9m53s kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.4.2" already present on machine
Normal Created 9m53s kubelet Created container elastic-internal-suspend
Normal Started 9m52s kubelet Started container elastic-internal-suspend
Normal Pulled 8m59s (x4 over 9m52s) kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.4.2" already present on machine
Normal Created 8m59s (x4 over 9m52s) kubelet Created container elasticsearch
Normal Started 8m59s (x4 over 9m51s) kubelet Started container elasticsearch
Warning BackOff 4m44s (x24 over 9m42s) kubelet Back-off restarting failed container
I am sure that nfs server is working since I am able to use the mount directory
cd /var/nfs/
and create a file
touch text.
I can see this file in the nfs server so I guess is an issue with elastic config.
