Starting ES with TLS options

NewmazN24 · September 13, 2019, 5:55pm

Fresh install, I didn't modify any of my file except elasticsearch.yml. I do not know what I need to do, did wrong.

Java is installed as well.

Addon to ES yaml

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/bin/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/bin/elastic-certificates.p12

Tried

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

Error

/usr/share/elasticsearch/bin/elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-09-13T13:51:21,752][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.2.jar:7.3.2]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.2.jar:7.3.2]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.2.jar:7.3.2]

ikakavas · September 13, 2019, 7:49pm

This is unrelated to TLS ,the error you get is self explanatory:

org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root

Do not run elasticsearch as root

NewmazN24 · September 16, 2019, 12:53pm

Hello,

So I have tried without being root. What I faced is permission denied, therefore I need to do sudo /usr/share/elasticsearch/bin/elasticsearch / sudo systemctl start elasticsearch. What gets me again, an issue.

Since I haven't touched any other config, I am clueless as of why I would face that problem. I have launched ES as root before, however the TLS / SSL wasn't added to config file.

mbelaloui · September 16, 2019, 1:41pm

Hello NewmazN24,

Can you get us the result of your ls -la /usr/share/elasticsearch/bin/
I think this can be the elasticsearch service doesn't have the permission to read the certificate.

Try to chown -R elasticsearch:root /usr/share/elasticsearch/*

Mehdi.

NewmazN24 · September 16, 2019, 1:49pm

Thanks Medhi.

Before chown

[root@localhost ~]# ls -la /usr/share/elasticsearch/bin
total 18468
drwxr-xr-x. 2 root root 4096 Sep 16 08:36 .
drwxr-xr-x. 8 root root 201 Sep 16 09:16 ..
-rwxr-xr-x. 1 root root 1954 Sep 6 10:55 elasticsearch
-rwxr-xr-x. 1 root root 491 Sep 6 10:45 elasticsearch-certgen
-rwxr-xr-x. 1 root root 483 Sep 6 10:45 elasticsearch-certutil
-rwxr-xr-x. 1 root root 982 Sep 6 10:55 elasticsearch-cli
-rwxr-xr-x. 1 root root 433 Sep 6 10:45 elasticsearch-croneval
-rwxr-xr-x. 1 root root 2149 Sep 6 10:55 elasticsearch-env
-rwxr-xr-x. 1 root root 121 Sep 6 10:55 elasticsearch-keystore
-rwxr-xr-x. 1 root root 440 Sep 6 10:45 elasticsearch-migrate
-rwxr-xr-x. 1 root root 126 Sep 6 10:55 elasticsearch-node
-rwxr-xr-x. 1 root root 172 Sep 6 10:55 elasticsearch-plugin
-rwxr-xr-x. 1 root root 431 Sep 6 10:45 elasticsearch-saml-metadata
-rwxr-xr-x. 1 root root 438 Sep 6 10:45 elasticsearch-setup-passwords
-rwxr-xr-x. 1 root root 118 Sep 6 10:55 elasticsearch-shard
-rwxr-xr-x. 1 root root 427 Sep 6 10:45 elasticsearch-sql-cli
-rwxr-xr-x. 1 root root 18828739 Sep 6 10:45 elasticsearch-sql-cli-7.3.2.jar
-rwxr-xr-x. 1 root root 426 Sep 6 10:45 elasticsearch-syskeygen
-rwxr-xr-x. 1 root root 426 Sep 6 10:45 elasticsearch-users
-rwxr-xr-x. 1 root root 346 Sep 6 10:45 x-pack-env
-rwxr-xr-x. 1 root root 354 Sep 6 10:45 x-pack-security-env
-rwxr-xr-x. 1 root root 353 Sep 6 10:45 x-pack-watcher-env
[root@localhost ~]# ls -la /usr/share/elasticsearch
total 544
drwxr-xr-x. 8 root root 201 Sep 16 09:16 .
drwxr-xr-x. 77 root root 4096 Sep 16 08:36 ..
drwxr-xr-x. 2 root root 4096 Sep 16 08:36 bin
drwxr-xr-x. 2 root root 66 Sep 16 09:16 certs
-rw-------. 1 root root 3451 Sep 16 09:04 elastic-certificates.p12
-rw-------. 1 root root 2527 Sep 16 08:42 elastic-stack-ca.p12
drwxr-xr-x. 8 root root 96 Sep 16 08:36 jdk
drwxr-xr-x. 3 root root 4096 Sep 16 08:36 lib
-rw-r--r--. 1 root root 13675 Sep 6 10:38 LICENSE.txt
drwxr-xr-x. 33 root root 4096 Sep 16 08:36 modules
-rw-rw-r--. 1 root root 502598 Sep 6 10:45 NOTICE.txt
drwxr-xr-x. 2 root root 6 Sep 6 10:55 plugins
-rw-r--r--. 1 root root 8500 Sep 6 10:38 README.textile

After chown

[root@localhost ~]# ls -la /usr/share/elasticsearch
total 544
drwxr-xr-x. 8 root root 201 Sep 16 09:16 .
drwxr-xr-x. 77 root root 4096 Sep 16 08:36 ..
drwxr-xr-x. 2 elasticsearch root 4096 Sep 16 08:36 bin
drwxr-xr-x. 2 elasticsearch root 66 Sep 16 09:16 certs
-rw-------. 1 elasticsearch root 3451 Sep 16 09:04 elastic-certificates.p12
-rw-------. 1 elasticsearch root 2527 Sep 16 08:42 elastic-stack-ca.p12
drwxr-xr-x. 8 elasticsearch root 96 Sep 16 08:36 jdk
drwxr-xr-x. 3 elasticsearch root 4096 Sep 16 08:36 lib
-rw-r--r--. 1 elasticsearch root 13675 Sep 6 10:38 LICENSE.txt
drwxr-xr-x. 33 elasticsearch root 4096 Sep 16 08:36 modules
-rw-rw-r--. 1 elasticsearch root 502598 Sep 6 10:45 NOTICE.txt
drwxr-xr-x. 2 elasticsearch root 6 Sep 6 10:55 plugins
-rw-r--r--. 1 elasticsearch root 8500 Sep 6 10:38 README.textile
[root@localhost ~]# ls -la /usr/share/elasticsearch/bin
total 18468
drwxr-xr-x. 2 elasticsearch root 4096 Sep 16 08:36 .
drwxr-xr-x. 8 root root 201 Sep 16 09:16 ..
-rwxr-xr-x. 1 elasticsearch root 1954 Sep 6 10:55 elasticsearch
-rwxr-xr-x. 1 elasticsearch root 491 Sep 6 10:45 elasticsearch-certgen
-rwxr-xr-x. 1 elasticsearch root 483 Sep 6 10:45 elasticsearch-certutil
-rwxr-xr-x. 1 elasticsearch root 982 Sep 6 10:55 elasticsearch-cli
-rwxr-xr-x. 1 elasticsearch root 433 Sep 6 10:45 elasticsearch-croneval
-rwxr-xr-x. 1 elasticsearch root 2149 Sep 6 10:55 elasticsearch-env
-rwxr-xr-x. 1 elasticsearch root 121 Sep 6 10:55 elasticsearch-keystore
-rwxr-xr-x. 1 elasticsearch root 440 Sep 6 10:45 elasticsearch-migrate
-rwxr-xr-x. 1 elasticsearch root 126 Sep 6 10:55 elasticsearch-node
-rwxr-xr-x. 1 elasticsearch root 172 Sep 6 10:55 elasticsearch-plugin
-rwxr-xr-x. 1 elasticsearch root 431 Sep 6 10:45 elasticsearch-saml-metadata
-rwxr-xr-x. 1 elasticsearch root 438 Sep 6 10:45 elasticsearch-setup-passwords
-rwxr-xr-x. 1 elasticsearch root 118 Sep 6 10:55 elasticsearch-shard
-rwxr-xr-x. 1 elasticsearch root 427 Sep 6 10:45 elasticsearch-sql-cli
-rwxr-xr-x. 1 elasticsearch root 18828739 Sep 6 10:45 elasticsearch-sql-cli-7.3.2.jar
-rwxr-xr-x. 1 elasticsearch root 426 Sep 6 10:45 elasticsearch-syskeygen
-rwxr-xr-x. 1 elasticsearch root 426 Sep 6 10:45 elasticsearch-users
-rwxr-xr-x. 1 elasticsearch root 346 Sep 6 10:45 x-pack-env
-rwxr-xr-x. 1 elasticsearch root 354 Sep 6 10:45 x-pack-security-env
-rwxr-xr-x. 1 elasticsearch root 353 Sep 6 10:45 x-pack-watcher-env

NewmazN24 · September 16, 2019, 1:53pm

It's still failing. I am trying to play with my config file.

Started

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

Tried

xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/certs/elastic-certificates.p12

xpack.security.transport.ssl.keystore.path: {ES_HOME}*/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: *{ES_HOME}/elastic-certificates.p12

mbelaloui · September 16, 2019, 2:04pm

Hello NewmazN24,

I apologize for not being more specific the chown -R elasticsearch:root /usr/share/elasticsearch/* is not a good practice i think, a best practice is to chown the certificate only so you don't change the predefined parameters.

You should also put your certificates in /etc/elasticsearch/ so the elasticsearch service can access it.

did you define a password when you create your certificate ?

can we have all yourelasticsearch.yml, the result of your sudo /usr/share/elasticsearch/bin/elasticsearch-keystore list and all the elasticsearch log 'starting when you restart your service'

Mehdi.

NewmazN24 · September 16, 2019, 2:15pm

Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)

Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:310)
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: at org.elasticsearch.cli.Command.main(Command.java:90)

Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,307 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingF
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,313 main ERROR Null object returned for RollingFile in Appenders.
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,313 main ERROR Null object returned for RollingFile in Appenders.
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,314 main ERROR Null object returned for RollingFile in Appenders.
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,314 main ERROR Null object returned for RollingFile in Appenders.
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,314 main ERROR Null object returned for RollingFile in Appenders.
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,315 main ERROR Null object returned for RollingFile in Appenders.
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,315 main ERROR Null object returned for RollingFile in Appenders.
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,315 main ERROR Null object returned for RollingFile in Appenders.
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,316 main ERROR Null object returned for RollingFile in Appenders.
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,316 main ERROR Unable to locate appender "rolling" for logger config "root"
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,316 main ERROR Unable to locate appender "rolling_old" for logger config "root"
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,317 main ERROR Unable to locate appender "index_indexing_slowlog_rolling_old" for logger config "index.indexing.slowlog.index"
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,321 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.indexing.slowlog.index"
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,321 main ERROR Unable to locate appender "audit_rolling" for logger config "org.elasticsearch.xpack.security.audit.logfile.LoggingAu
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,321 main ERROR Unable to locate appender "index_search_slowlog_rolling_old" for logger config "index.search.slowlog"
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,323 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,324 main ERROR Unable to locate appender "deprecation_rolling_old" for logger config "org.elasticsearch.deprecation"
Sep 16 10:10:58 localhost.localdomain elasticsearch[7092]: 2019-09-16 10:10:58,324 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
Sep 16 10:11:02 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Sep 16 10:11:02 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Sep 16 10:11:02 localhost.localdomain systemd[1]: elasticsearch.service failed.

NewmazN24 · September 16, 2019, 2:17pm

Certificate are under /etc/elasticsearch.

ES yaml file has only that configure, everything else is as is.

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

NewmazN24 · September 16, 2019, 2:22pm

You won't be able to help me properly if I do not tell you all you need.

Run on CentOS7
Only Elasticsearch is installed
Nothing has been modified, except lines in elasticsearch.yml

Ran those command

elasticsearch-certutil cert --ca elastic-stack-ca.p12 (With a password).
- Modified elasticsearch.yml file after running the command.
elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

ikakavas · September 16, 2019, 2:41pm

I would imagine that this permission denied was due to the fact that you copied the elastic-certificates.p12 files in the configuration folder but those were owned by root and so elasticsearch could not read them. The solution for this would be to make the keystore files readable by elasticsearch, not to run elasticsearch as root.

I think your current state is because of permission changes that you have tried , could that be the case ?

Is that a new installation , or the previous ones we were dealing with ? i.e. have you performed any ownership changes for files and directories or not ?

Which user did you run this commands as ? Root ? This will affect who owns the produced keystore files and whether elasticsearch can read them.

NewmazN24 · September 16, 2019, 5:20pm

@ikakavas It finally ran. For some reason, CHOWN didn't apply properly on my repertory.

@mbelaloui Thanks for your points !

mbelaloui · September 16, 2019, 9:23pm

Happy To Help.

system · October 14, 2019, 9:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.