X-Pack Security Not Working

Elastic Stack Elasticsearch
1

Hey Guys,

I followed the instructions https://www.elastic.co/downloads/x-pack and configured X-PACK. I set all the passwords and then tried to go to Kibana and got

Login is currently disabled. Administrators should consult the Kibana logs for more details.

I then searched through some forums and found out that xpack.security.enabled: true needs to be set in both kibana.yml and elasticsearch.yml so I did that and restarted both. Now, ElasticSearch is hosed and this is what it says. Basically that it doesnt know the setting. What am I doing wrong?

[2017-12-05T13:31:33,318][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalArgumentException: unknown setting [xpack.security.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:293) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:256) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:135) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:330) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.0.0.jar:6.0.0]
[2017-12-05T13:31:33,319][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [xpack.security.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:134) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.0.0.jar:6.0.0]
Caused by: java.lang.IllegalArgumentException: unknown setting [xpack.security.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:293) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:256) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:135) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:330) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) ~[elasticsearch-6.0.0.jar:6.0.0]
... 6 more

1 Like
2

It really doesn't look like x-pack is installed on your elasicsearch cluster.

Can you follow these steps:

  • Remove the xpack.security.enabled line from config/elasticsearch.yml (you don't need it, security is enabled by default once x-pack is installed)
  • Run this from your elasticsearch installation directory (which will show which plugins are installed on the filesystem)
bin/elasticsearch-plugin list --verbose
  • Start Elasticsearch
  • And then run this (which will show which plugins are actually running on the cluster)
curl 'http://localhost:9200/_cat/plugins?v&pretty'
  • And run this (which tests the most minimal of x-pack functionality)
curl 'http://localhost:9200/_xpack?categories=features&pretty'
1 Like
3

Tim, you were right, for whatever reason x-pack was not installed on elasticsearch (I ran a full installation and it did not error out).

I tried it again and it installed OK. But now I get this when I try to start Elastic. I can disable the ML pieces but what does it mean by X-Pack is not supported?

[2017-12-06T08:08:49,067][ERROR][o.e.b.Bootstrap ] Exception
org.elasticsearch.ElasticsearchException: X-Pack is not supported and Machine Learning is not available for [linux-x86]; you can use the other X-Pack features (unsupported) by setting xpack.ml.enabled: false in elasticsearch.yml
at org.elasticsearch.xpack.ml.MachineLearningFeatureSet.isRunningOnMlPlatform(MachineLearningFeatureSet.java:112) ~[?:?]
at org.elasticsearch.xpack.ml.MachineLearningFeatureSet.isRunningOnMlPlatform(MachineLearningFeatureSet.java:103) ~[?:?]
at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:319) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.createComponents(XPackPlugin.java:332) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$6(Node.java:399) ~[elasticsearch-6.0.0.jar:6.0.0]
at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:1.8.0_144]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:1.8.0_144]
at org.elasticsearch.node.Node.(Node.java:402) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.0.0.jar:6.0.0]
[2017-12-06T08:08:49,070][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[X-Pack is not supported and Machine Learning is not available for [linux-x86]; you can use the other X-Pack features (unsupported) by setting xpack.ml.enabled: false in elasticsearch.yml]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:134) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.0.0.jar:6.0.0]
Caused by: org.elasticsearch.ElasticsearchException: X-Pack is not supported and Machine Learning is not available for [linux-x86]; you can use the other X-Pack features (unsupported) by setting xpack.ml.enabled: false in elasticsearch.yml
at org.elasticsearch.xpack.ml.MachineLearningFeatureSet.isRunningOnMlPlatform(MachineLearningFeatureSet.java:112) ~[?:?]
at org.elasticsearch.xpack.ml.MachineLearningFeatureSet.isRunningOnMlPlatform(MachineLearningFeatureSet.java:103) ~[?:?]
at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:319) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.createComponents(XPackPlugin.java:332) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$6(Node.java:399) ~[elasticsearch-6.0.0.jar:6.0.0]
at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:1.8.0_144]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:1.8.0_144]
at org.elasticsearch.node.Node.(Node.java:402) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) ~[elasticsearch-6.0.0.jar:6.0.0]
... 6 more

4

May be that my JVM is 32bit. Trying to install a 64bit JVM.

5

I installed the 64-bit JVM and Elastic is now on, however Kibana has difficulty connecting to Elastic. I have configured the xpack plugin setting in kibana as well as set the user kibana and password generated in the xpack process.

This is what I see in the Kibana logs, looks like Kibana may be having difficulty authenticating?

{"type":"log","@timestamp":"2017-12-06T13:45:09Z","tags":["status","ui settings","error"],"pid":24492,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}

6

HI @adwaitjoshi,

It does look like that it can't connect. If you share a little more of your logs above that we will be able to verify this. In the meantime, please run the commands that @TimV shared with you above and share the output so that we can see the state of your installation now and help you move on.

7

First command

[root@DESKTOP-7MAG1N2 /]# /usr/share/elasticsearch/bin/elasticsearch-plugin list --verbose
Plugins directory: /usr/share/elasticsearch/plugins
x-pack

  • Plugin information:
    Name: x-pack
    Description: Elasticsearch Expanded Pack Plugin
    Version: 6.0.0
    Native Controller: true
    Requires Keystore: true
  • Classname: org.elasticsearch.xpack.XPackPlugin
    [root@DESKTOP-7MAG1N2 /]#

Second command

[root@DESKTOP-7MAG1N2 /]# curl 'http://localhost:9200/_cat/plugins?v&pretty'
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_cat/plugins?v&pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_cat/plugins?v&pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
}

Third command

[root@DESKTOP-7MAG1N2 /]# curl 'http://localhost:9200/_xpack?categories=features&pretty'
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_xpack?categories=features&pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_xpack?categories=features&pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
}

8

Great, so X-Pack is installed and security is enabled. You mention

I have configured the xpack plugin setting in kibana as well as set the user kibana and password generated in the xpack process.

So I assume that you have run the setup-passwords as stated in the instructions and set or generated the passwords for your builtin users and then you have used the password for the kibana system user in kibana.yml :

elasticsearch.password: <the_password_you_set_for_kibana>

Can you verify that those are correct by running :

curl -u kibana 'http://localhost:9200/_cat/plugins?v&pretty' and entering your kibana system user password when prompted ?

9

[root@DESKTOP-7MAG1N2 /]# curl -u kibana 'http://localhost:9200/_cat/plugins?v&pretty'
Enter host password for user 'kibana':
name component version
node-1 x-pack 6.0.0
[root@DESKTOP-7MAG1N2 /]#

10

So the password is correct. Assuming that you kibana.yml also says :

elasticsearch.username: kibana

it shouldn't be an authentication problem. Can you share some more of your kibana logs and additionally any portions of kibana.yml that you might have changed after installation ?

Finally, just to verify you have installed X-Pack for kibana too, please check :

bin/kibana-plugin list

11

[root@DESKTOP-7MAG1N2 ~]# /usr/share/kibana/bin/kibana-plugin list
x-pack@6.0.0

[root@DESKTOP-7MAG1N2 ~]#

12

Can you share some more of your kibana logs please ?

13

Ok I uploaded the file here http://dataseers.us/temp/kibana.stdout

14

Do those logs help in anyway? I cannot seem to figure it out at all.

15

FYI I am able to connect to the license

Enter host password for user 'elastic':
{
  "license" : {
    "status" : "active",
    "uid" : "e34c9585-2984-49c6-9296-6a9d7198543a",
    "type" : "trial",
    "issue_date" : "2017-12-06T13:27:10.211Z",
    "issue_date_in_millis" : 1512566830211,
    "expiry_date" : "2018-01-05T13:27:10.211Z",
    "expiry_date_in_millis" : 1515158830211,
    "max_nodes" : 1000,
    "issued_to" : "dataseers",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1
  }
}

I can connect using the kibana user as well.

[root@DESKTOP-7MAG1N2 /]# curl -u kibana 'http://localhost:9200/_cat/plugins?v&pretty'
Enter host password for user 'kibana':
name component version
node-1 x-pack 6.0.0
[root@DESKTOP-7MAG1N2 /]#

However in the kibana logs the elastic plugin still says red. But it wont give me any more log details.

Anyways, I dont know if I am over complicating with x-pack. All I honestly care about is basic security with x-pack, I failed to read the detailed documentation which indicates I may need a license? Thankfully the license has a 30 day trial so it has not expired yet, but can I still do user authentication with basic x-pack or I need to purchase a license for this?

16

Hi again,

Can we get some logs from Elasticsearch also ?

Your issue seems to be similar to this one where indices are locked because of low disk space on the host, but your logs will verify this.

If this is the case you can attempt to unlock them by:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
17

That unlocking the indices seemed to have worked. Thanks a lot. However, why would the indices lock I don't have a full disk? The most used mount is 22%. Its rather odd.

18

So after the x-Pack trial license has expired, do I still have the ability to do basic user authentication or do I have to buy a separate license for that?

19

I don't have any good guesses right now. Maybe if you kept any Elasticsearch logs from when you were seeing the issue, we can get to the bottom of it.

As you can see in our subscriptions page, security is not enabled with the Basic license.

20

Is xpack the only way to enable security on kibana and elastic?