X-Pack Security Not Working

adwaitjoshi · December 5, 2017, 6:59pm

Hey Guys,

I followed the instructions https://www.elastic.co/downloads/x-pack and configured X-PACK. I set all the passwords and then tried to go to Kibana and got

Login is currently disabled. Administrators should consult the Kibana logs for more details.

I then searched through some forums and found out that xpack.security.enabled: true needs to be set in both kibana.yml and elasticsearch.yml so I did that and restarted both. Now, ElasticSearch is hosed and this is what it says. Basically that it doesnt know the setting. What am I doing wrong?

[2017-12-05T13:31:33,318][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalArgumentException: unknown setting [xpack.security.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:293) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:256) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:135) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:330) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.0.0.jar:6.0.0]
[2017-12-05T13:31:33,319][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [xpack.security.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:134) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.0.0.jar:6.0.0]
Caused by: java.lang.IllegalArgumentException: unknown setting [xpack.security.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:293) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:256) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:135) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:330) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) ~[elasticsearch-6.0.0.jar:6.0.0]
... 6 more

TimV · December 6, 2017, 12:41am

It really doesn't look like x-pack is installed on your elasicsearch cluster.

Can you follow these steps:

Remove the xpack.security.enabled line from config/elasticsearch.yml (you don't need it, security is enabled by default once x-pack is installed)
Run this from your elasticsearch installation directory (which will show which plugins are installed on the filesystem)

bin/elasticsearch-plugin list --verbose

Start Elasticsearch
And then run this (which will show which plugins are actually running on the cluster)

curl 'http://localhost:9200/_cat/plugins?v&pretty'

And run this (which tests the most minimal of x-pack functionality)

curl 'http://localhost:9200/_xpack?categories=features&pretty'

adwaitjoshi · December 6, 2017, 1:14pm

Tim, you were right, for whatever reason x-pack was not installed on elasticsearch (I ran a full installation and it did not error out).

I tried it again and it installed OK. But now I get this when I try to start Elastic. I can disable the ML pieces but what does it mean by X-Pack is not supported?

[2017-12-06T08:08:49,067][ERROR][o.e.b.Bootstrap ] Exception
org.elasticsearch.ElasticsearchException: X-Pack is not supported and Machine Learning is not available for [linux-x86]; you can use the other X-Pack features (unsupported) by setting xpack.ml.enabled: false in elasticsearch.yml
at org.elasticsearch.xpack.ml.MachineLearningFeatureSet.isRunningOnMlPlatform(MachineLearningFeatureSet.java:112) ~[?:?]
at org.elasticsearch.xpack.ml.MachineLearningFeatureSet.isRunningOnMlPlatform(MachineLearningFeatureSet.java:103) ~[?:?]
at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:319) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.createComponents(XPackPlugin.java:332) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$6(Node.java:399) ~[elasticsearch-6.0.0.jar:6.0.0]
at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:1.8.0_144]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:1.8.0_144]
at org.elasticsearch.node.Node.(Node.java:402) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.0.0.jar:6.0.0]
[2017-12-06T08:08:49,070][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[X-Pack is not supported and Machine Learning is not available for [linux-x86]; you can use the other X-Pack features (unsupported) by setting xpack.ml.enabled: false in elasticsearch.yml]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:134) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.0.0.jar:6.0.0]
Caused by: org.elasticsearch.ElasticsearchException: X-Pack is not supported and Machine Learning is not available for [linux-x86]; you can use the other X-Pack features (unsupported) by setting xpack.ml.enabled: false in elasticsearch.yml
at org.elasticsearch.xpack.ml.MachineLearningFeatureSet.isRunningOnMlPlatform(MachineLearningFeatureSet.java:112) ~[?:?]
at org.elasticsearch.xpack.ml.MachineLearningFeatureSet.isRunningOnMlPlatform(MachineLearningFeatureSet.java:103) ~[?:?]
at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:319) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.createComponents(XPackPlugin.java:332) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$6(Node.java:399) ~[elasticsearch-6.0.0.jar:6.0.0]
at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:1.8.0_144]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:1.8.0_144]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:1.8.0_144]
at org.elasticsearch.node.Node.(Node.java:402) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) ~[elasticsearch-6.0.0.jar:6.0.0]
... 6 more

adwaitjoshi · December 6, 2017, 1:16pm

May be that my JVM is 32bit. Trying to install a 64bit JVM.

adwaitjoshi · December 6, 2017, 1:47pm

I installed the 64-bit JVM and Elastic is now on, however Kibana has difficulty connecting to Elastic. I have configured the xpack plugin setting in kibana as well as set the user kibana and password generated in the xpack process.

This is what I see in the Kibana logs, looks like Kibana may be having difficulty authenticating?

{"type":"log","@timestamp":"2017-12-06T13:45:09Z","tags":["status","ui settings","error"],"pid":24492,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}

ikakavas · December 6, 2017, 2:00pm

HI @adwaitjoshi,

It does look like that it can't connect. If you share a little more of your logs above that we will be able to verify this. In the meantime, please run the commands that @TimV shared with you above and share the output so that we can see the state of your installation now and help you move on.

adwaitjoshi · December 6, 2017, 2:15pm

First command

[root@DESKTOP-7MAG1N2 /]# /usr/share/elasticsearch/bin/elasticsearch-plugin list --verbose
Plugins directory: /usr/share/elasticsearch/plugins
x-pack

Plugin information:
Name: x-pack
Description: Elasticsearch Expanded Pack Plugin
Version: 6.0.0
Native Controller: true
Requires Keystore: true

Classname: org.elasticsearch.xpack.XPackPlugin
[root@DESKTOP-7MAG1N2 /]#

Second command

[root@DESKTOP-7MAG1N2 /]# curl 'http://localhost:9200/_cat/plugins?v&pretty'
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_cat/plugins?v&pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_cat/plugins?v&pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
}

Third command

[root@DESKTOP-7MAG1N2 /]# curl 'http://localhost:9200/_xpack?categories=features&pretty'
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_xpack?categories=features&pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_xpack?categories=features&pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
}

ikakavas · December 6, 2017, 2:31pm

Great, so X-Pack is installed and security is enabled. You mention

I have configured the xpack plugin setting in kibana as well as set the user kibana and password generated in the xpack process.

So I assume that you have run the setup-passwords as stated in the instructions and set or generated the passwords for your builtin users and then you have used the password for the kibana system user in kibana.yml :

elasticsearch.password: <the_password_you_set_for_kibana>

Can you verify that those are correct by running :

curl -u kibana 'http://localhost:9200/_cat/plugins?v&pretty' and entering your kibana system user password when prompted ?

adwaitjoshi · December 6, 2017, 2:41pm

[root@DESKTOP-7MAG1N2 /]# curl -u kibana 'http://localhost:9200/_cat/plugins?v&pretty'
Enter host password for user 'kibana':
name component version
node-1 x-pack 6.0.0
[root@DESKTOP-7MAG1N2 /]#

ikakavas · December 6, 2017, 3:01pm

So the password is correct. Assuming that you kibana.yml also says :

elasticsearch.username: kibana

it shouldn't be an authentication problem. Can you share some more of your kibana logs and additionally any portions of kibana.yml that you might have changed after installation ?

Finally, just to verify you have installed X-Pack for kibana too, please check :

bin/kibana-plugin list

adwaitjoshi · December 6, 2017, 4:17pm

[root@DESKTOP-7MAG1N2 ~]# /usr/share/kibana/bin/kibana-plugin list
x-pack@6.0.0

[root@DESKTOP-7MAG1N2 ~]#

ikakavas · December 6, 2017, 4:20pm

Can you share some more of your kibana logs please ?

adwaitjoshi · December 6, 2017, 7:42pm

Ok I uploaded the file here http://dataseers.us/temp/kibana.stdout

adwaitjoshi · December 7, 2017, 4:21pm

Do those logs help in anyway? I cannot seem to figure it out at all.

adwaitjoshi · December 7, 2017, 9:12pm

FYI I am able to connect to the license

Enter host password for user 'elastic':
{
  "license" : {
    "status" : "active",
    "uid" : "e34c9585-2984-49c6-9296-6a9d7198543a",
    "type" : "trial",
    "issue_date" : "2017-12-06T13:27:10.211Z",
    "issue_date_in_millis" : 1512566830211,
    "expiry_date" : "2018-01-05T13:27:10.211Z",
    "expiry_date_in_millis" : 1515158830211,
    "max_nodes" : 1000,
    "issued_to" : "dataseers",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1
  }
}

I can connect using the kibana user as well.

[root@DESKTOP-7MAG1N2 /]# curl -u kibana 'http://localhost:9200/_cat/plugins?v&pretty'
Enter host password for user 'kibana':
name component version
node-1 x-pack 6.0.0
[root@DESKTOP-7MAG1N2 /]#

However in the kibana logs the elastic plugin still says red. But it wont give me any more log details.

Anyways, I dont know if I am over complicating with x-pack. All I honestly care about is basic security with x-pack, I failed to read the detailed documentation which indicates I may need a license? Thankfully the license has a 30 day trial so it has not expired yet, but can I still do user authentication with basic x-pack or I need to purchase a license for this?

ikakavas · December 8, 2017, 7:13am

Hi again,

Can we get some logs from Elasticsearch also ?

Your issue seems to be similar to this one where indices are locked because of low disk space on the host, but your logs will verify this.

If this is the case you can attempt to unlock them by:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

adwaitjoshi · December 8, 2017, 1:02pm

That unlocking the indices seemed to have worked. Thanks a lot. However, why would the indices lock I don't have a full disk? The most used mount is 22%. Its rather odd.

adwaitjoshi · December 8, 2017, 1:15pm

So after the x-Pack trial license has expired, do I still have the ability to do basic user authentication or do I have to buy a separate license for that?

ikakavas · December 8, 2017, 3:35pm

I don't have any good guesses right now. Maybe if you kept any Elasticsearch logs from when you were seeing the issue, we can get to the bottom of it.

As you can see in our subscriptions page, security is not enabled with the Basic license.

adwaitjoshi · December 8, 2017, 8:12pm

Is xpack the only way to enable security on kibana and elastic?

Topic		Replies	Views
Setting xpack.security.enabled = true Elasticsearch elastic-stack-security	8	25489	June 29, 2019
Issue with Installing X-Pack on ELK - Security Exception Error Elasticsearch	6	1326	February 8, 2018
Elastic Security Issues Elasticsearch elastic-stack-security	2	243	October 11, 2023
Kibana login username & password Kibana	16	35907	January 12, 2017
Elasticsearch is restarting like a loop after I remove x-pack security Elasticsearch elastic-stack-security	4	898	October 22, 2019

X-Pack Security Not Working

Related Topics