Elasticsearch 7.3.2 not starting with security enabled

Elastic Stack Elasticsearch
1

Dear all,

Elasticsearch:

[root@fchiorascu-2 ~]# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: fchiorascu-efk
node.name: fchiorascu-2
path.data: /data/elasticsearch/fchiorascu-2
path.logs: /var/log/elasticsearch
network.host: ip2
discovery.zen.ping.unicast.hosts: [ip2,ip3,ip4]
discovery.zen.minimum_master_nodes: 2
http.cors.enabled: true
http.cors.allow-origin: "*"
bootstrap.memory_lock: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12
#xpack.security.http.ssl.enabled: true
#xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
#xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12
#xpack.security.http.ssl.client_authentication: optional

Kibana:

[root@fchiorascu-1 ~]# cat /etc/kibana/kibana.yml
server.port: 5601
server.host: "ip1"
elasticsearch.ssl.verificationMode: none
logging.dest: /var/log/kibana/kibana.log
logging.verbose: false
server.name: fchiorascu-1
elasticsearch.hosts:

Output Elasticsearch:

[root@fchiorascu-2 ~]# tail -f /var/log/elasticsearch/fchiorascu-efk.log
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.node.Node.(Node.java:314) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.node.Node.(Node.java:258) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.2.jar:7.3.2]
... 6 more

> I read all the articles from the internet and also official documentation but the thing is that I have a configuration of 1 x kibana + 3 x elasticsearch (with the role of master and node in the same time). 

How will be better to approach the situation with the certificates?

Normally you generate the certificate on master with certutil and copy on nodes, but here all are masters/nodes in the same time.

Kind Regards,

2

Your log is incomplete
You can check here https://www.elastic.co/guide/en/elasticsearch/reference/7.3/bootstrap-checks.html

3

Yes, you are right I did not touch the bootstrap for all the aspects.
The log is below, a part of it.

[root@fchiorascu-3 ~]# cat /var/log/elasticsearch/fchiorascu-efk.log
[2019-09-20T00:00:19,897][INFO ][o.e.e.NodeEnvironment ] [fchiorascu-3] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [36.6gb], net total_space [39.9gb], types [rootfs]
[2019-09-20T00:00:19,920][INFO ][o.e.e.NodeEnvironment ] [fchiorascu-3] heap size [1.9gb], compressed ordinary object pointers [true]
[2019-09-20T00:00:19,923][INFO ][o.e.n.Node ] [fchiorascu-3] node name [fchiorascu-3], node ID [GNnCLgsVS76mwXagxksrKA], cluster name [fchiorascu-efk]
[2019-09-20T00:00:19,923][INFO ][o.e.n.Node ] [fchiorascu-3] version[7.3.2], pid[18115], build[default/rpm/1c1faf1/2019-09-06T14:40:30.409026Z], OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.2/12.0.2+10]
[2019-09-20T00:00:19,924][INFO ][o.e.n.Node ] [fchiorascu-3] JVM home [/usr/share/elasticsearch/jdk]
[2019-09-20T00:00:19,924][INFO ][o.e.n.Node ] [fchiorascu-3] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-6462549703277393978, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=1073741824, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2019-09-20T00:00:20,618][ERROR][o.e.b.Bootstrap ] [fchiorascu-3] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.node.Node.(Node.java:314) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.node.Node.(Node.java:258) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.3.2.jar:7.3.2]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.3.2.jar:7.3.2]
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.2.jar:7.3.2]
... 15 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1133) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$2(SSLService.java:426) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1333) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:423) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:119) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.(XPackPlugin.java:146) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.2.jar:7.3.2]
... 15 more

4

Based on the logs is complaining about some permissions.

Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/config/elastic-certificates.p12
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:373) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:424) ~[?:?]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
at java.nio.file.Files.newInputStream(Files.java:158) ~[?:?]
at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:87) ~[?:?]
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1133) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$2(SSLService.java:426) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1333) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:423) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:119) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.(XPackPlugin.java:146) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.2.jar:7.3.2]
... 15 more

5

You need to make sure that the user as which the elasticsearch process runs has access to read the file. What is the output of

ls -l /etc/elasticsearch/config/elastic-certificates.p12

and

ls -l /etc/elasticsearch/config

?

6

you still do it on one master node and copy over to all other nodes. that is what I have done.
all my nodes are master/data

7

@ikakavas thank you for the details.
I've figured out that I've missed something there. SOLVED now.

I was looking after,

[root@fchiorascu-2 ~]# ls -ld /etc/elasticsearch /etc/elasticsearch/config/
drwxr-s---. 4 elasticsearch elasticsearch 4096 Oct 18 14:38 /etc/elasticsearch
drwxr-sr-x. 2 elasticsearch elasticsearch 38 Oct 17 22:10 /etc/elasticsearch/config/

[root@fchiorascu-2 ~]# ls -ltrha /etc/elasticsearch/config/elastic-certificates.p12
-rw-------. 1 elasticsearch elasticsearch 3.4K Oct 17 22:10 /etc/elasticsearch/config/elastic-certificates.p12

it was root elasticsearch --> /etc/elasticsearch/config/elastic-certificates.p12 <<

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.