# X-pack not working on new ELK stack **URL:** \ **Category:** Elasticsearch\ **Created:** [October 24, 2017, 3:58pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093 "2017-10-24T15:58:05Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![carlfriedrichgauss](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@carlfriedrichgauss](https://discuss.elastic.co/u/carlfriedrichgauss)\ **Post date:** [October 24, 2017, 3:58pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/1 "2017-10-24T15:58:05Z")
ES and Kibana on 5.6.3 ... Everything in the cluster seems to be working fine, except in the browser I get the error: Login is currently disabled because the license could not be determined. Please check that Elasticsearch has the X-Pack plugin installed and is reachable, then refresh this page. I see other users getting this error, but only on upgrades. This is a fresh install. I even tried the hotfix here, tho I know I'm on 5.6.3 ([Upgrade issue with Elastic Stack 5.6.0, workaround option until fix is available](https://discuss.elastic.co/t/upgrade-issue-with-elastic-stack-5-6-0-workaround-option-until-fix-is-available/100595)) The ES logs seem to load x-pack just fine: [2017-10-24T11:42:08,799][INFO][o.e.p.PluginsService] [vnDDbdu] loaded plugin [x-pack] The license seems to be active. [root@elkhost elasticsearch]# curl -GET -u elastic:changeme '[http://localhost:9200/\_xpack/license](http://localhost:9200/_xpack/license)' { "license" : { "status" : "active", ... "type" : "trial", ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 24, 2017, 5:12pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/2 "2017-10-24T17:12:05Z")
FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats (and now also APM!) feels left out 😉 What do the logs for Kibana show? ---
**Author:** ![carlfriedrichgauss](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@carlfriedrichgauss](https://discuss.elastic.co/u/carlfriedrichgauss)\ **Post date:** [October 24, 2017, 5:19pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/3 "2017-10-24T17:19:53Z")
I think it may be a failure to connect to ES [root@elkhost kibana]# grep -i x kibana.stdout ["status","plugin:xpack\_main@5.6.3","error"],"pid":29320,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at [https://localhost:9200](https://localhost:9200).","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2017-10-24T14:55:30Z","tags":["reporting","warning"],"pid":29320,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"} EDIT: Adding this [root@elkhost kibana]# curl [https://localhost:9200](https://localhost:9200) curl: (35) SSL received a record that exceeded the maximum permissible length. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 24, 2017, 6:16pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/4 "2017-10-24T18:16:23Z")
That's odd, do you have a proxy somewhere that might be causing this? ---
**Author:** ![carlfriedrichgauss](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@carlfriedrichgauss](https://discuss.elastic.co/u/carlfriedrichgauss)\ **Post date:** [October 24, 2017, 7:11pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/5 "2017-10-24T19:11:23Z")
Nope, not proxy is configured. Whole stack is on a single RHEL VM. Any config you need to see? elasticsearch.yml and kibana.yml are all I have modified, and that I have only modified lightly. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 24, 2017, 8:25pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/6 "2017-10-24T20:25:55Z")
How are you configuring SSL on Elasticsearch? ---
**Author:** ![carlfriedrichgauss](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@carlfriedrichgauss](https://discuss.elastic.co/u/carlfriedrichgauss)\ **Post date:** [October 24, 2017, 8:40pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/7 "2017-10-24T20:40:56Z")
All I've done is added these lines: xpack.ssl.key: /etc/elasticsearch/x-pack/elk.key xpack.ssl.certificate: /etc/elasticsearch/x-pack/elk.crt xpack.security.transport.ssl.enabled: true xpack.security.http.ssl.enabled: true Those certs are generated and signed by an external CA. If I try to add a cacerts file, ES complains that it "failed to initialize a TrustManagerFactory" I can curl against https if I add the "--insecure" options. Makes me think it's not running TLS/SSL at all actually. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [October 25, 2017, 12:06am UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/8 "2017-10-25T00:06:01Z")
> [@carlfriedrichgauss](#): > > I can curl against https if I add the "--insecure" options. Makes me think it's not running TLS/SSL at all actually. No, the `--insecure` option enables TLS without proper checking of CA trust. It implies that your elasticsearch node is running with TLS enabled but something is wrong with the certificates. Some things you can try: - double check the elasticsearch logs for any SSL related messages - run curl with `--verbose` or `--trace-ascii %` - check the cert with `openssl` and see if anything looks out of place. Try something like: - check the SSL connection with `openssl s_client`. Try something like: ---
**Author:** ![carlfriedrichgauss](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@carlfriedrichgauss](https://discuss.elastic.co/u/carlfriedrichgauss)\ **Post date:** [October 25, 2017, 4:33pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/9 "2017-10-25T16:33:44Z")
I think the problem line (from the documentation) is this: `xpack.ssl.certificate_authorities: ["/home/es/config/x-pack/ca.crt"]` In an attempt to make this as generic as possible, I used Elastic's `certgen` tool. The line in my config is: `xpack.ssl.certificate_authorities: /etc/elasticsearch/x-pack/ca/ca.crt` And this is causing all my SSL traces in the logs. There are a lot of error components, but I think this is the one that matters: `IllegalArgumentException[parsed an unsupported object [X509CertificateHolder]];` I think it has to do with how I'm formatting that line. The zip file `certgen` created contained 2 keys (node.key and ca.key) and 2 crts (node.crt and ca.crt). How am I formatting this line wrong?? I tried initially with the exact information from the instructions ([https://www.elastic.co/guide/en/x-pack/current/ssl-tls.html](https://www.elastic.co/guide/en/x-pack/current/ssl-tls.html)) and that failed. I keep trying variations, but they're all failing. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [October 26, 2017, 2:06am UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/10 "2017-10-26T02:06:13Z")
> [@carlfriedrichgauss](#): > > ``` > IllegalArgumentException[parsed an unsupported object [X509CertificateHolder]]; > > ``` That's definitely a problem, can you provide a bit more of the stack trace? Nothing in your configuration stands out as a problem, but if you can give me a few more details on the error then I may be able to track it down. My guess is that the problem is that one of your `.key` files (probably `/etc/elasticsearch/x-pack/elk.key`) actually contains a certificate rather than a key. I don't think `certgen` would have done that - perhaps you accidentally copied the wrong file? Can you check the first line of each of those files? The ones named `*.crt` should start with ```auto -----BEGIN CERTIFICATE----- ``` And the ones named `*.key` should start with: ```auto -----BEGIN RSA PRIVATE KEY----- ``` ---
**Author:** ![carlfriedrichgauss](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@carlfriedrichgauss](https://discuss.elastic.co/u/carlfriedrichgauss)\ **Post date:** [October 26, 2017, 2:37pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/11 "2017-10-26T14:37:59Z")
I wanted it to be a typo so bad, but alas, it was not. Key and Crt files are as the should be, 5 dashes and all. Could there be an issue with my yaml? I would think it would thrown a yaml parsing error tho, and not a java trace. Full trace: [2017-10-25T12:30:03,317][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]]; nested: InvocationTargetException; nested: ElasticsearchException[failed to initialize a KeyManagerFactory]; nested: IllegalArgumentException[parsed an unsupported object [X509CertificateHolder]]; at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.3.jar:5.6.3] Caused by: org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:434) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:387) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.plugins.PluginsService.(PluginsService.java:140) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.node.Node.(Node.java:313) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.3.jar:5.6.3] ... 6 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0\_151] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:423) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:387) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.plugins.PluginsService.(PluginsService.java:140) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.node.Node.(Node.java:313) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.3.jar:5.6.3] ... 6 more Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a KeyManagerFactory at org.elasticsearch.xpack.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:72) ~[?:?] at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:395) ~[?:?] at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:416) ~[?:?] at org.elasticsearch.xpack.ssl.SSLService.(SSLService.java:79) ~[?:?] at org.elasticsearch.xpack.XPackPlugin.(XPackPlugin.java:216) ~[?:?] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0\_151] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:423) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:387) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.plugins.PluginsService.(PluginsService.java:140) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.node.Node.(Node.java:313) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.3.jar:5.6.3] ... 6 more I can remove this error from the logs only by removing the "xpack.ssl.certificate\_authorities" directive. It works, but I have to use the --insecure option, as noted before. That's what is making me think this is the source of the problem. [root@elkhost x-pack]# openssl x509 -in /etc/elasticsearch/x-pack/ca/ca.crt -text Certificate: ... Signature Algorithm: sha256WithRSAEncryption Issuer: CN=Elastic Certificate Tool Autogenerated CA ... Subject: CN=Elastic Certificate Tool Autogenerated CA [root@elkhost x-pack]# openssl x509 -in /etc/elasticsearch/x-pack/elkhost/elkhost.crt -text Certificate: ,... Issuer: CN=Elastic Certificate Tool Autogenerated CA ... Subject: CN=elkhost ---
**Author:** ![carlfriedrichgauss](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@carlfriedrichgauss](https://discuss.elastic.co/u/carlfriedrichgauss)\ **Post date:** [October 26, 2017, 2:38pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/12 "2017-10-26T14:38:17Z")
Rest of the log (just append to the end, entry was too long) Caused by: java.lang.IllegalArgumentException: parsed an unsupported object [X509CertificateHolder] at org.elasticsearch.xpack.ssl.CertUtils.innerReadPrivateKey(CertUtils.java:274) ~[?:?] at org.elasticsearch.xpack.ssl.CertUtils.readPrivateKey(CertUtils.java:239) ~[?:?] at org.elasticsearch.xpack.ssl.PEMKeyConfig.readPrivateKey(PEMKeyConfig.java:78) ~[?:?] at org.elasticsearch.xpack.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:64) ~[?:?] at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:395) ~[?:?] at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:416) ~[?:?] at org.elasticsearch.xpack.ssl.SSLService.(SSLService.java:79) ~[?:?] at org.elasticsearch.xpack.XPackPlugin.(XPackPlugin.java:216) ~[?:?] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0\_151] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:423) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:387) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.plugins.PluginsService.(PluginsService.java:140) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.node.Node.(Node.java:313) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.3.jar:5.6.3] ... 6 more ---
**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\ **Post date:** [October 26, 2017, 4:06pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/13 "2017-10-26T16:06:49Z")
If you look in your `elk.key` file is there any section other than the private key one? ``` -----BEGIN RSA PRIVATE KEY----- -----END RSA PRIVATE KEY----- ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 23, 2017, 4:07pm UTC](https://discuss.elastic.co/t/x-pack-not-working-on-new-elk-stack/105093/14 "2017-11-23T16:07:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.