Trobling with HTTPS configuration

thewasta · June 26, 2019, 11:47am

First: My versión is 6.6.1.
My enviroment: 3 nodes in 3 servers (all local)
Node Master: 192.168.1.142
Node Slave1: 192.168.1.144
Node Slave2: 192.168.1.146

I have been following this Guide from Elastic Doc, everything works fine with transport.ssl, all services works, still haven tried to create user because I think it's useless if the connection to server is not https. Then I tried the next step, activate HTTPs, at first I created all certificates with passwords using command:

bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --name elasticmaster --ip 192.168.1.142

Same command but changing name and ip for others nodes, but got error "AccessDenied wrong password" in ElasticSearch logs, so I decide to configurate it again but without password for all certificates, but still the same error. Tried to create a new elasticsearch.keyword with 'create' flag and again created certificates, but nothing all time same error.
And of course I added the key for each node running command (in the case when I used password, if I didnt use, I removed this keystore).

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

It's the same error for all nodes. By the way, the password which I have to type when I run the previous command is the node's password, right? And... How may I reset all? Just removing all files (certs files)?
Any advice?
Thanks,

TimV · June 26, 2019, 12:16pm

That doesn't sounds like an Elasticsearch error.

Can you copy-and-paste the actual error from the logs? Paraphrased error messages are very hard to work with.

thewasta · June 26, 2019, 12:44pm

It's very extensed file. I'm still working on it. nevermind. Here the log from Master Node:

[2019-06-26T14:39:44,712][INFO ][o.e.e.NodeEnvironment ] [elasticmaster] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [83.6gb], net total_space [97.9gb], types [ext4]
[2019-06-26T14:39:44,716][INFO ][o.e.e.NodeEnvironment ] [elasticmaster] heap size [5.9gb], compressed ordinary object pointers [true]
[2019-06-26T14:39:45,044][INFO ][o.e.n.Node ] [elasticmaster] node name [elasticmaster], node ID [RhMwLe4rQO-oKoBjUxHFSw]

java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:364) ~[?:?]
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elasticmaster.p12
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:608) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:550) ~[elasticsearch-6.6.1.jar:6.6.1]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:364) ~[?:?]
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elasticmaster.p12
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]

Can't share the entire output because of limit characters.
Thanks

thewasta · June 26, 2019, 12:49pm

More info: I created the certificate for the nodes from Node Master, then I copied this file to their respective server with scp command.

ikakavas · June 26, 2019, 1:38pm

You've cut the output right where the actual cause would be shown. If it doesn't fit in one message you can split it in two and please format the logs you paste correctly using the </> button or triple backticks (```) as it is much harder to read as plain text. You can use the preview panel on the right to see how it looks like before you post it

thewasta · June 26, 2019, 1:42pm

[2019-06-26T15:39:19,297][INFO ][o.e.e.NodeEnvironment    ] [elasticmaster] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [83.6gb], net total_space [97.9gb], types [ext4]
[2019-06-26T15:39:19,302][INFO ][o.e.e.NodeEnvironment    ] [elasticmaster] heap size [5.9gb], compressed ordinary object pointers [true]
[2019-06-26T15:39:19,423][INFO ][o.e.n.Node               ] [elasticmaster] node name [elasticmaster], node ID [RhMwLe4rQO-oKoBjUxHFSw]
[2019-06-26T15:39:19,424][INFO ][o.e.n.Node               ] [elasticmaster] version[6.6.1], pid[24480], build[default/deb/1fd8f69/2019-02-13T17:10:04.160291Z], OS[Linux/4.15.0-50-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_201/25.201-b09]
[2019-06-26T15:39:19,424][INFO ][o.e.n.Node               ] [elasticmaster] JVM arguments [-Xms6g, -Xmx6g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-6278558871907171410, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb]
[2019-06-26T15:39:20,019][ERROR][o.e.b.Bootstrap          ] [elasticmaster] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:608) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:550) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:465) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:157) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:337) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.6.1.jar:6.6.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.6.1.jar:6.6.1]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]

thewasta · June 26, 2019, 1:43pm

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_201]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:599) ~[elasticsearch-6.6.1.jar:6.6.1]
	... 15 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
	at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:364) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_201]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:421) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:96) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_201]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:599) ~[elasticsearch-6.6.1.jar:6.6.1]
	... 15 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elasticmaster.p12
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_201]
	at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_201]
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_201]
	at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_201]
	at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:87) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:364) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_201]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:421) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:96) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_201]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:599) ~[elasticsearch-6.6.1.jar:6.6.1]
	... 15 more
[2019-06-26T15:39:20,030][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticmaster] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.1.jar:6.6.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:608) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:550) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:465) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:157) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:337) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_201]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:599) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:550) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:465) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:157) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:337) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]

thewasta · June 26, 2019, 1:43pm

	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
	... 6 more

Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
	at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:364) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_201]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:421) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:96) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_201]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:599) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:550) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:465) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:157) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:337) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
	... 6 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elasticmaster.p12
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_201]
	at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_201]
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_201]
	at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_201]
	at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:87) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:364) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_201]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:421) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:96) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_201]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:599) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:550) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:465) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:157) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:337) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
	... 6 more

ikakavas · June 26, 2019, 1:45pm

Elasticsearch complains that it can't read the file /etc/elasticsearch/certs/elasticmaster.p12 . What are the permissions for that file ? Can the user that elasticsearch is running as read it ?

thewasta · June 26, 2019, 1:46pm

Configured:

chown -R root:elasticsearch certs
chmod 660 certs/*

ikakavas · June 26, 2019, 1:48pm

Under which user is elasticsearch running ? is this user a member of the elasticsearch group ?

thewasta · June 26, 2019, 1:52pm

Well... I can't answer this question, because I don't know how to know it...
Actually I'm doing all with root, but mainly user is "administrator".
One second and I will search how to know it.

thewasta · June 26, 2019, 1:58pm

I feel disappointed but don't know how to answer the question
May you help me?

ikakavas · June 26, 2019, 2:07pm

How did you install Elasticsearch?

You can try running

ps ax o pid,user,group,command | grep elasticsearch

and share the output

thewasta · June 26, 2019, 2:13pm

Slave 1

31066 elastic+ elastic+ /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-15520827159542884468 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -XX:UseAVX=2 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
31160 elastic+ elastic+ /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
31251 root     root     grep --color=auto elasticsearch

Master

24628 elastic+ elastic+ /usr/bin/java -Xms6g -Xmx6g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-2016259871112841528 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
24716 elastic+ elastic+ /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
24819 root     root     grep --color=auto elasticsearch

Slave 2

 4445 elastic+ elastic+ /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-1115501696674566959 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -XX:UseAVX=2 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
 4542 elastic+ elastic+ /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
 4589 root     root     grep --color=auto elasticsearch

What I don't understand is... If it is permision problem, why can access to file if I comment/delete the lines, if both use same folder/files

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

ikakavas · June 26, 2019, 2:20pm

Apologies, but I don't quite understand what you are saying here, can you please rephrase this?

Your certs folder and files is owned by user root and the elasticsearch group

chown -R root:elasticsearch certs

and only the owner and members of the elasticsearch group can read and write to the file.

chmod 660 certs/*

Elasticserach is running under the elastic system user as seen by the ps output. Is that elastic user member of your elasticsearch group? if not, you either need to give them the membership or make sure that the p12 file is owned by the elastic user.

thewasta · June 26, 2019, 2:29pm

This line/configuration in elasticsearch.yml is using the file .p12 and If I don't add the HTTPS configuration Elasticsearch runs well. Aren't both using same User/Group to access to same file?

xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12

If I run command groups only have root and with command cut -d: -f1 /etc/passwd there is elasticsearch user, should I run this:

sudo usermod -a -G root elasticsearch

TimV · June 27, 2019, 3:11am

The error message you provided says:

java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elasticmaster.p12

That's a different file.

Can you please provide the output of:

ls -la  /etc/elasticsearch/certs/

thewasta · June 27, 2019, 12:59pm

Don't check the name, the first name is from 'guide' elastic-certificadesis equal to elasticmaster/elasticslave1/elasticslave2.

This is the output

drwxr-sr-x 2 root elasticsearch 4096 jun 26 15:31 .
drwxr-s--- 3 root elasticsearch 4096 jun 26 16:53 ..
-rw-rw---- 1 root elasticsearch 3493 jun 26 15:31 elasticmaster.p12
-rw-rw---- 1 root elasticsearch 3485 jun 26 15:31 elasticslave1.p12
-rw-rw---- 1 root elasticsearch 3493 jun 26 15:31 elasticslave2.p12
-rw-rw---- 1 root elasticsearch 2527 jun 26 15:29 elastic-stack-ca.p12

This output is from Node Master. In the server Node Master is where I created all certs and then I copied the file to their respective Node using scp command. Example:

cd /etc/elasticsearch/certs
scp elasticslave1.p12 admin@192.168.1.144:/home/admin/

ikakavas · June 27, 2019, 1:07pm

We can't not check the name. You have to give us exact information if we are to help you. When you use different filenames in every post it's very very hard for us to follow you and to find out what goes on in your deployment.

Share the exact xpack.security.transport.ssl. configuration from elasticsearch.yml from all of your nodes, without changing any filenames please.