# Elasticsearch 7.3.2 not starting with security enabled

**URL:** <https://discuss.elastic.co/t/elasticsearch-7-3-2-not-starting-with-security-enabled/200458>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [September 20, 2019, 4:07pm UTC](https://discuss.elastic.co/t/elasticsearch-7-3-2-not-starting-with-security-enabled/200458 "2019-09-20T16:07:54Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![fchiorascu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fchiorascu/32/83962_2.png) [@fchiorascu](https://discuss.elastic.co/u/fchiorascu)\
**Post date:** [September 20, 2019, 4:07pm UTC](https://discuss.elastic.co/t/elasticsearch-7-3-2-not-starting-with-security-enabled/200458/1 "2019-09-20T16:07:54Z")

</div>

Dear all,

**Elasticsearch:**

> [root@fchiorascu-2 ~]# cat /etc/elasticsearch/elasticsearch.yml  
> cluster.name: fchiorascu-efk  
> node.name: fchiorascu-2  
> path.data: /data/elasticsearch/fchiorascu-2  
> path.logs: /var/log/elasticsearch  
> network.host: ip2  
> discovery.zen.ping.unicast.hosts: [ip2,ip3,ip4]  
> discovery.zen.minimum\_master\_nodes: 2  
> http.cors.enabled: true  
> http.cors.allow-origin: "\*"  
> bootstrap.memory\_lock: true  
> xpack.security.enabled: true  
> xpack.security.transport.ssl.enabled: true  
> xpack.security.transport.ssl.verification\_mode: certificate  
> xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12  
> xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12  
> #xpack.security.http.ssl.enabled: true  
> #xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12  
> #xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12  
> #xpack.security.http.ssl.client\_authentication: optional

**Kibana:**

> [root@fchiorascu-1 ~]# cat /etc/kibana/kibana.yml  
> server.port: 5601  
> server.host: "ip1"  
> elasticsearch.ssl.verificationMode: none  
> logging.dest: /var/log/kibana/kibana.log  
> logging.verbose: false  
> server.name: fchiorascu-1  
> elasticsearch.hosts:
> 
> - [http://ip2:9200](http://ip2:9200)
> - [http://ip3:9200](http://ip3:9200)
> - [http://ip4:9200](http://ip4:9200)  
> #elasticsearch.username: "elastic"  
> #elasticsearch.password: "changeme"  
> xpack.monitoring.enabled: true  
> xpack.security.enabled: true

**Output Elasticsearch:**

> [root@fchiorascu-2 ~]# tail -f /var/log/elasticsearch/fchiorascu-efk.log  
> at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.node.Node.(Node.java:314) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.node.Node.(Node.java:258) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.2.jar:7.3.2]  
> ... 6 more

```
> I read all the articles from the internet and also official documentation but the thing is that I have a configuration of 1 x kibana + 3 x elasticsearch (with the role of master and node in the same time). 

```

> ```
> How will be better to approach the situation with the certificates?
> 
> ```
> 
> Normally you generate the certificate on master with certutil and copy on nodes, but here all are masters/nodes in the same time.

Kind Regards,

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [September 20, 2019, 4:17pm UTC](https://discuss.elastic.co/t/elasticsearch-7-3-2-not-starting-with-security-enabled/200458/2 "2019-09-20T16:17:51Z")

</div>

Your log is incomplete  
You can check here [https://www.elastic.co/guide/en/elasticsearch/reference/7.3/bootstrap-checks.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/bootstrap-checks.html)

---

<div class="post-metadata">

**Author:** ![fchiorascu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fchiorascu/32/83962_2.png) [@fchiorascu](https://discuss.elastic.co/u/fchiorascu)\
**Post date:** [September 20, 2019, 4:27pm UTC](https://discuss.elastic.co/t/elasticsearch-7-3-2-not-starting-with-security-enabled/200458/3 "2019-09-20T16:27:46Z")

</div>

Yes, you are right I did not touch the bootstrap for all the aspects.  
The log is below, a part of it.

> [root@fchiorascu-3 ~]# cat /var/log/elasticsearch/fchiorascu-efk.log  
> [2019-09-20T00:00:19,897][INFO][o.e.e.NodeEnvironment] [fchiorascu-3] using [1] data paths, mounts [[/ (rootfs)]], net usable\_space [36.6gb], net total\_space [39.9gb], types [rootfs]  
> [2019-09-20T00:00:19,920][INFO][o.e.e.NodeEnvironment] [fchiorascu-3] heap size [1.9gb], compressed ordinary object pointers [true]  
> [2019-09-20T00:00:19,923][INFO][o.e.n.Node] [fchiorascu-3] node name [fchiorascu-3], node ID [GNnCLgsVS76mwXagxksrKA], cluster name [fchiorascu-efk]  
> [2019-09-20T00:00:19,923][INFO][o.e.n.Node] [fchiorascu-3] version[7.3.2], pid[18115], build[default/rpm/1c1faf1/2019-09-06T14:40:30.409026Z], OS[Linux/3.10.0-862.14.4.el7.x86\_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.2/12.0.2+10]  
> [2019-09-20T00:00:19,924][INFO][o.e.n.Node] [fchiorascu-3] JVM home [/usr/share/elasticsearch/jdk]  
> [2019-09-20T00:00:19,924][INFO][o.e.n.Node] [fchiorascu-3] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-6462549703277393978, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs\_err\_pid%p.log, -Xlog:gc\*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=1073741824, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled\_jdk=true]  
> [2019-09-20T00:00:20,618][ERROR][o.e.b.Bootstrap] [fchiorascu-3] Exception  
> java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]  
> at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.node.Node.(Node.java:314) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.node.Node.(Node.java:258) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.3.2.jar:7.3.2]  
> at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.3.2.jar:7.3.2]  
> at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.3.2.jar:7.3.2]  
> at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.3.2.jar:7.3.2]  
> Caused by: java.lang.reflect.InvocationTargetException  
> at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]  
> at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]  
> at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]  
> at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]  
> at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]  
> at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.2.jar:7.3.2]  
> ... 15 more  
> Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory  
> at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]  
> at java.util.HashMap.computeIfAbsent(HashMap.java:1133) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$2(SSLService.java:426) ~[?:?]  
> at java.util.HashMap.forEach(HashMap.java:1333) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:423) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:119) ~[?:?]  
> at org.elasticsearch.xpack.core.XPackPlugin.(XPackPlugin.java:146) ~[?:?]  
> at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]  
> at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]  
> at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]  
> at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]  
> at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]  
> at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.2.jar:7.3.2]  
> ... 15 more

---

<div class="post-metadata">

**Author:** ![fchiorascu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fchiorascu/32/83962_2.png) [@fchiorascu](https://discuss.elastic.co/u/fchiorascu)\
**Post date:** [September 20, 2019, 4:29pm UTC](https://discuss.elastic.co/t/elasticsearch-7-3-2-not-starting-with-security-enabled/200458/4 "2019-09-20T16:29:19Z")

</div>

Based on the logs is complaining about some permissions.

> **Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/config/elastic-certificates.p12**  
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]  
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]  
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]  
> at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]  
> at java.nio.file.Files.newByteChannel(Files.java:373) ~[?:?]  
> at java.nio.file.Files.newByteChannel(Files.java:424) ~[?:?]  
> at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]  
> at java.nio.file.Files.newInputStream(Files.java:158) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:87) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]  
> at java.util.HashMap.computeIfAbsent(HashMap.java:1133) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$2(SSLService.java:426) ~[?:?]  
> at java.util.HashMap.forEach(HashMap.java:1333) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:423) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:119) ~[?:?]  
> at org.elasticsearch.xpack.core.XPackPlugin.(XPackPlugin.java:146) ~[?:?]  
> at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]  
> at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]  
> at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]  
> at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]  
> at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]  
> at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.2.jar:7.3.2]  
> ... 15 more

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 24, 2019, 8:45pm UTC](https://discuss.elastic.co/t/elasticsearch-7-3-2-not-starting-with-security-enabled/200458/5 "2019-09-24T20:45:16Z")

</div>

> [@fchiorascu](#):
>
> Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/config/elastic-certificates.p12

You need to make sure that the user as which the elasticsearch process runs has access to read the file. What is the output of

```auto
ls -l /etc/elasticsearch/config/elastic-certificates.p12

```

and

```auto
ls -l /etc/elasticsearch/config

```

?

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [September 24, 2019, 9:25pm UTC](https://discuss.elastic.co/t/elasticsearch-7-3-2-not-starting-with-security-enabled/200458/6 "2019-09-24T21:25:39Z")

</div>

you still do it on one master node and copy over to all other nodes. that is what I have done.  
all my nodes are master/data

---

<div class="post-metadata">

**Author:** ![fchiorascu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fchiorascu/32/83962_2.png) [@fchiorascu](https://discuss.elastic.co/u/fchiorascu)\
**Post date:** [October 18, 2019, 11:46am UTC](https://discuss.elastic.co/t/elasticsearch-7-3-2-not-starting-with-security-enabled/200458/7 "2019-10-18T11:46:34Z")

</div>

@ikakavas thank you for the details.  
I've figured out that I've missed something there. SOLVED now.

I was looking after,

[root@fchiorascu-2 ~]# ls -ld /etc/elasticsearch /etc/elasticsearch/config/  
drwxr-s---. 4 elasticsearch elasticsearch 4096 Oct 18 14:38 /etc/elasticsearch  
drwxr-sr-x. 2 elasticsearch elasticsearch 38 Oct 17 22:10 /etc/elasticsearch/config/

[root@fchiorascu-2 ~]# ls -ltrha /etc/elasticsearch/config/elastic-certificates.p12  
-rw-------. 1 elasticsearch elasticsearch 3.4K Oct 17 22:10 /etc/elasticsearch/config/elastic-certificates.p12

> > it was root elasticsearch --\> /etc/elasticsearch/config/elastic-certificates.p12 \<\<

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 15, 2019, 5:51pm UTC](https://discuss.elastic.co/t/elasticsearch-7-3-2-not-starting-with-security-enabled/200458/10 "2019-11-15T17:51:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.