Elasticsearch with Helm Charts (Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes)

bishwajitsamanta1689 · March 28, 2019, 10:07am

I am unable to create Elasticsearch by Helm charts. Need some guidance. I am a layman so please explain me in details if possible.

root@kubemaster:~/kube_dir/elasticsearch# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

elasticsearch-master-elasticsearch-master-0 Bound elasticsearch-vol-3 10Gi RWO manual 10m
elasticsearch-master-elasticsearch-master-1 Bound elasticsearch-vol-2 10Gi RWO manual 10m
elasticsearch-master-elasticsearch-master-2 Bound elasticsearch-vol-1 10Gi RWO manual 10m

root@kubemaster:~/kube_dir/elasticsearch# kubectl get pv

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
elasticsearch-vol-1 10Gi RWO Retain Bound default/elasticsearch-master-elasticsearch-master-2 manual 12m
elasticsearch-vol-2 10Gi RWO Retain Bound default/elasticsearch-master-elasticsearch-master-1 manual 17m
elasticsearch-vol-3 10Gi RWO Retain Bound default/elasticsearch-master-elasticsearch-master-0 manual 17m

root@kubemaster:~/kube_dir/elasticsearch# kubectl get pods

NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 0/1 CrashLoopBackOff 6 11m
elasticsearch-master-1 0/1 CrashLoopBackOff 6 11m
elasticsearch-master-2 0/1 Pending 0 11m

root@kubemaster:~/kube_dir/elasticsearch# kubectl logs elasticsearch-master-0

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [2019-03-28T10:04:30,179][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-master-0] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.2.jar:6.6.2] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.2.jar:6.6.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.2.jar:6.6.2] Caused by: java.lang.IllegalStateException: Failed to create node environment at org.elasticsearch.node.Node.<init>(Node.java:298) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.2.jar:6.6.2] ... 6 more Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:385) ~[?:?] at java.nio.file.Files.createDirectory(Files.java:689) ~[?:?] at java.nio.file.Files.createAndCheckIsDirectory(Files.java:796) ~[?:?] at java.nio.file.Files.createDirectories(Files.java:782) ~[?:?] at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:270) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:203) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.node.Node.<init>(Node.java:295) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.2.jar:6.6.2] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.2.jar:6.6.2] ... 6 more

root@kubemaster:~/kube_dir/elasticsearch# ls -lrth /data1/
total 0
root@kubemaster:~/kube_dir/elasticsearch# ls -lrth /data2/
total 16K
drwxrwxrwx 2 apcuser apcuser 16K Mar 27 03:08 lost+found
root@kubemaster:~/kube_dir/elasticsearch# ls -lrth /data3/
total 16K
drwxrwxrwx 2 apcuser apcuser 16K Mar 27 03:08 lost+found
root@kubemaster:~/kube_dir/elasticsearch# ls -lrth /data3/

Abhilash_B · March 28, 2019, 10:32am

Please take out time and format the logs for easier readability. You can use common markdown to format your posts.

bishwajitsamanta1689 · March 29, 2019, 1:58am

Can any one reply please..

michael.russell · March 29, 2019, 5:08am

Which helm chart are you using to deploy this cluster? Based on the hostnames I think it is the elastic helm chart but it would be nice to confirm.

Can you also give me the following details:

The values.yaml or configuration flags you are using to deploy the cluster
Which Kubernetes version and provider you are using. (E.g. 1.12 on GKE)
Which user the Elasticsearch process is running as inside the docker container

Based only on the error message you have given I think you might be using a Kubernetes version like Openshift which changes the overrides the user that the container runs as. If this is the case one of the steps you will need to take is setting the fsGroup to the user that Elasticsearch is running as if this is not the default user with id 1000.

bishwajitsamanta1689 · April 1, 2019, 9:39am

Thanks for your reply michael,
I am using Kubernetes Version : 1.13.4. And i am using Elastic Helm Chart. i am following the steps mentioned in the document.
https://github.com/elastic/helm-charts/tree/master/elasticsearch.
I havent modified any value.yml file i just run those 2 commands.
helm repo add elastic https://helm.elastic.co
and
helm install --name elasticsearch elastic/elasticsearch --version 6.6.2-alpha1
my default system user is apcuser, so i am using the same user and have given permission to the storage mount place with the same user and 777 permissions.

i am using Local Disk for Persistence Storage. I just mounted one hard drive as use it for persistence volume.
It would be nice if you can help me with this. I am not much learned person about kubernetes helm. so i am trying my hands on to it, and what other than elasticsearch i can try my hands on , atleast i can get some support by doing this first.

michael.russell · April 1, 2019, 12:04pm

Thanks for the info! It sounds like you will need to modify the default storageClaimTemplate that comes with the chart.

You will need to replace standard with the storage class name for your local disks. You can find this by running kubectl get storageclass. An example of how this looks for me:

$ kubectl get storageclass
NAME                 PROVISIONER            AGE
standard (default)   kubernetes.io/gce-pd   124d

Could you also provide the output of:

kubectl get events

bishwajitsamanta1689 · April 1, 2019, 12:23pm

root@kubemaster:~# kubectl get events
LAST SEEN TYPE REASON KIND MESSAGE

56s Warning BackOff Pod Back-off restarting failed container
20m Normal Pulled Pod Container image "docker.elastic.co/elasticsearch/elasticsearch:6.6.2" already present on machine
53s Warning BackOff Pod Back-off restarting failed container
105s Warning FailedScheduling Pod 0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't satisfy existing pods anti-affinity rules.
7m51s Normal Scheduled Pod Successfully assigned default/init-demo to kubeclient
3m55s Normal Pulling Pod pulling image "busybox"
3m31s Normal Pulled Pod Successfully pulled image "busybox"
3m31s Normal Created Pod Created container
4m50s Normal Started Pod Started container
4m7s Warning BackOff Pod Back-off restarting failed container
2m42s Normal Scheduled Pod Successfully assigned default/init-demo to kubeclient
33s Normal Pulling Pod pulling image "busybox"
9s Normal Pulled Pod Successfully pulled image "busybox"
9s Normal Created Pod Created container
9s Normal Started Pod Started container
3s Warning BackOff Pod Back-off restarting failed container

This is my Persistent Volume Yaml which i am creating.

apiVersion: v1
kind: PersistentVolume
metadata:
name: elasticsearch-vol-1
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
storageClassName: manual
hostPath:
path: /data1

And this is the Persistent Volume Claim i am creating as it remains in Pending stage so i delete them manually and create persistent volume claim .

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: elasticsearch-master-elasticsearch-master-0
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
root@kubemaster:~/kube_dir/elasticsearch# kubectl get storageclass No resources found.

michael.russell · April 1, 2019, 1:06pm

This is my Persistent Volume Yaml which i am creating.

This is not something you should be creating. Instead this should be set as the storageClaimTemplate in your values file for the helm chart.

105s Warning FailedScheduling Pod 0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't satisfy existing pods anti-affinity rules.

Are you running a single node Kubernetes cluster? This isn't really recommended and should only be done if you are wanting to develop the chart locally using something like minikube or docker-for-mac. To get it working you will need to set soft anti affinity like in the minikube example in the repo.

Your values.yaml file should look something like this:

antiAffinity: "soft"

volumeClaimTemplate:
  storageClassName: manual
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

bishwajitsamanta1689 · April 1, 2019, 1:15pm

root@kubemaster:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubeclient Ready 4d5h v1.13.4
kubeclient2 Ready 4d5h v1.13.4
kubemaster Ready master 4d5h v1.13.4

My question is if i am not creating Persistent Volume, how this thing is going to work out.

root@kubemaster:~/kube_dir/charts/stable/elasticsearch# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 99G 0 part /
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 975M 0 part
sdb 8:16 0 30G 0 disk
├─sdb1 8:17 0 10G 0 part /data1
├─sdb2 8:18 0 10G 0 part /data2
└─sdb3 8:19 0 10G 0 part /data3
sdc 8:32 0 30G 0 disk
├─sdc1 8:33 0 10G 0 part /data4
├─sdc2 8:34 0 10G 0 part /data5
└─sdc3 8:35 0 10G 0 part /data6
sdd 8:48 0 30G 0 disk
├─sdd1 8:49 0 10G 0 part /data7
├─sdd2 8:50 0 10G 0 part /data8
└─sdd3 8:51 0 10G 0 part /data9
sr0 11:0 1 1024M 0 rom
loop0 7:0 0 91.1M 1 loop /snap/core/6531
loop1 7:1 0 8.8M 1 loop /snap/helm/85
loop2 7:2 0 89.3M 1 loop /snap/core/6673

can you guide me what should i do for getting this done.

bishwajitsamanta1689 · April 8, 2019, 10:39am

Can anyone assist me in resolving the issue ?

bishwajitsamanta1689 · April 12, 2019, 10:27am

i have tried the solution by adding volumeClaimTemplate, but it is not working. Can any one let me know how to install elasticsearch through helm in a local disk environment.

bishwajitsamanta1689 · April 15, 2019, 6:57am

Thanks everyone for their help. I finally resolved my issue.

system · May 13, 2019, 6:57am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.