X-pack not working on new ELK stack

carlfriedrichgauss · October 24, 2017, 3:58pm

ES and Kibana on 5.6.3 ... Everything in the cluster seems to be working fine, except in the browser I get the error:

Login is currently disabled because the license could not be determined. Please check that Elasticsearch has the X-Pack plugin installed and is reachable, then refresh this page.

I see other users getting this error, but only on upgrades. This is a fresh install. I even tried the hotfix here, tho I know I'm on 5.6.3 (Upgrade issue with Elastic Stack 5.6.0, workaround option until fix is available)

The ES logs seem to load x-pack just fine:

[2017-10-24T11:42:08,799][INFO ][o.e.p.PluginsService ] [vnDDbdu] loaded plugin [x-pack]

The license seems to be active.
[root@elkhost elasticsearch]# curl -GET -u elastic:changeme 'http://localhost:9200/_xpack/license'
{
"license" : {
"status" : "active",
...
"type" : "trial",

warkolm · October 24, 2017, 5:12pm

FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats (and now also APM!) feels left out

What do the logs for Kibana show?

carlfriedrichgauss · October 24, 2017, 5:19pm

I think it may be a failure to connect to ES

[root@elkhost kibana]# grep -i x kibana.stdout
["status","plugin:xpack_main@5.6.3","error"],"pid":29320,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at https://localhost:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-10-24T14:55:30Z","tags":["reporting","warning"],"pid":29320,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}

EDIT:
Adding this

[root@elkhost kibana]# curl https://localhost:9200
curl: (35) SSL received a record that exceeded the maximum permissible length.

warkolm · October 24, 2017, 6:16pm

That's odd, do you have a proxy somewhere that might be causing this?

carlfriedrichgauss · October 24, 2017, 7:11pm

Nope, not proxy is configured. Whole stack is on a single RHEL VM. Any config you need to see? elasticsearch.yml and kibana.yml are all I have modified, and that I have only modified lightly.

warkolm · October 24, 2017, 8:25pm

How are you configuring SSL on Elasticsearch?

carlfriedrichgauss · October 24, 2017, 8:40pm

All I've done is added these lines:

xpack.ssl.key: /etc/elasticsearch/x-pack/elk.key
xpack.ssl.certificate: /etc/elasticsearch/x-pack/elk.crt

xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true

Those certs are generated and signed by an external CA. If I try to add a cacerts file, ES complains that it "failed to initialize a TrustManagerFactory"

I can curl against https if I add the "--insecure" options. Makes me think it's not running TLS/SSL at all actually.

TimV · October 25, 2017, 12:06am

No, the --insecure option enables TLS without proper checking of CA trust. It implies that your elasticsearch node is running with TLS enabled but something is wrong with the certificates.

Some things you can try:

double check the elasticsearch logs for any SSL related messages
run curl with --verbose or --trace-ascii %
check the cert with openssl and see if anything looks out of place. Try something like:
```
openssl x509 -noout -text -in /etc/elasticsearch/x-pack/elk.crt
```
check the SSL connection with openssl s_client. Try something like:
```
openssl s_client -connect localhost:9200  -showcerts -tls1
```

carlfriedrichgauss · October 25, 2017, 4:33pm

I think the problem line (from the documentation) is this:

xpack.ssl.certificate_authorities: [ "/home/es/config/x-pack/ca.crt" ]

In an attempt to make this as generic as possible, I used Elastic's certgen tool. The line in my config is:

xpack.ssl.certificate_authorities: /etc/elasticsearch/x-pack/ca/ca.crt

And this is causing all my SSL traces in the logs. There are a lot of error components, but I think this is the one that matters:

IllegalArgumentException[parsed an unsupported object [X509CertificateHolder]];

I think it has to do with how I'm formatting that line. The zip file certgen created contained 2 keys (node.key and ca.key) and 2 crts (node.crt and ca.crt). How am I formatting this line wrong?? I tried initially with the exact information from the instructions (https://www.elastic.co/guide/en/x-pack/current/ssl-tls.html) and that failed. I keep trying variations, but they're all failing.

TimV · October 26, 2017, 2:06am

That's definitely a problem, can you provide a bit more of the stack trace?
Nothing in your configuration stands out as a problem, but if you can give me a few more details on the error then I may be able to track it down.

My guess is that the problem is that one of your .key files (probably /etc/elasticsearch/x-pack/elk.key) actually contains a certificate rather than a key.
I don't think certgen would have done that - perhaps you accidentally copied the wrong file?

Can you check the first line of each of those files?
The ones named *.crt should start with

-----BEGIN CERTIFICATE-----

And the ones named *.key should start with:

-----BEGIN RSA PRIVATE KEY-----

carlfriedrichgauss · October 26, 2017, 2:37pm

I wanted it to be a typo so bad, but alas, it was not. Key and Crt files are as the should be, 5 dashes and all. Could there be an issue with my yaml? I would think it would thrown a yaml parsing error tho, and not a java trace.

Full trace:

[2017-10-25T12:30:03,317][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]]; nested: InvocationTargetException; nested: ElasticsearchException[failed to initialize a KeyManagerFactory]; nested: IllegalArgumentException[parsed an unsupported object [X509CertificateHolder]];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.3.jar:5.6.3]
Caused by: org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:434) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:387) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:140) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.node.Node.(Node.java:313) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.3.jar:5.6.3]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_151]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:423) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:387) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:140) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.node.Node.(Node.java:313) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.3.jar:5.6.3]
... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a KeyManagerFactory
at org.elasticsearch.xpack.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:72) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:395) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:416) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.(SSLService.java:79) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.(XPackPlugin.java:216) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_151]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:423) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:387) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:140) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.node.Node.(Node.java:313) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.3.jar:5.6.3]
... 6 more

I can remove this error from the logs only by removing the "xpack.ssl.certificate_authorities" directive. It works, but I have to use the --insecure option, as noted before. That's what is making me think this is the source of the problem.

[root@elkhost x-pack]# openssl x509 -in /etc/elasticsearch/x-pack/ca/ca.crt -text
Certificate:
...
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Elastic Certificate Tool Autogenerated CA
...
Subject: CN=Elastic Certificate Tool Autogenerated CA

[root@elkhost x-pack]# openssl x509 -in /etc/elasticsearch/x-pack/elkhost/elkhost.crt -text
Certificate:
,...
Issuer: CN=Elastic Certificate Tool Autogenerated CA
...
Subject: CN=elkhost

carlfriedrichgauss · October 26, 2017, 2:38pm

Rest of the log (just append to the end, entry was too long)

Caused by: java.lang.IllegalArgumentException: parsed an unsupported object [X509CertificateHolder]
at org.elasticsearch.xpack.ssl.CertUtils.innerReadPrivateKey(CertUtils.java:274) ~[?:?]
at org.elasticsearch.xpack.ssl.CertUtils.readPrivateKey(CertUtils.java:239) ~[?:?]
at org.elasticsearch.xpack.ssl.PEMKeyConfig.readPrivateKey(PEMKeyConfig.java:78) ~[?:?]
at org.elasticsearch.xpack.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:64) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:395) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:416) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.(SSLService.java:79) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.(XPackPlugin.java:216) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_151]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:423) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:387) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:140) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.node.Node.(Node.java:313) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.3.jar:5.6.3]
... 6 more

jaymode · October 26, 2017, 4:06pm

If you look in your elk.key file is there any section other than the private key one?

-----BEGIN RSA PRIVATE KEY-----

-----END RSA PRIVATE KEY-----

system · November 23, 2017, 4:07pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.