Authentication for kibana, unknown setting xpack.security.enabled

I've had Elasticsearch and kibana as part of my docker-compose stack for some project for a while but now I need to add authentication for kibana and came across a strange problem. According to docs I need to set xpack.security.enabled param to true but it doesn't seems to work for me. I tried by adding it as environment in docker-compose file and as a config file attached to container - in both cases something is wrong.

This is relevant part of my configuration:

elasticsearch:
image: elasticsearch:7.8.0
container_name: elasticsearch
environment:
  - bootstrap.memory_lock=true
  - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
  - ELASTIC_USERNAME=elastic
  - ELASTIC_PASSWORD=MyPw123
  # - xpack.license.self_generated.type=trial 
  - xpack.security.enabled=true
  # - xpack.security.transport.ssl.enabled=true
ulimits:
  memlock:
    soft: -1
    hard: -1
#volumes:
#  - ${PWD}/cos.yaml:/etc/elasticsearch/elasticsearch.yml
ports:
  - 9200:9200
networks:
  - project-net

kibana:
image: kibana:7.8.0
environment:
  # ELASTICSEARCH_URL: http://elasticsearch:9200
  ELASTICSEARCH_HOSTS: http://elasticsearch:9200
  SERVER_BASEPATH: /logs
links:
  - "elasticsearch"
depends_on:
  - elasticsearch
networks:
  - project-net

When I set xpack.security.enabled=true I can't start my container. Logs below:

elasticsearch                        | {"type":"log","host":"e3933e9fe1c3","level":"WARN","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2020-11-05T15:07:04.437Z","logger":"o.e.c.l.LogConfigurator","timezone":"UTC","marker":"[localhost] ","log":"Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
elasticsearch                        |   /etc/elasticsearch/log4j2.properties"}
elasticsearch                        | {"type":"log","host":"e3933e9fe1c3","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2020-11-05T15:07:04.771Z","logger":"o.e.n.Node","timezone":"UTC","marker":"[localhost] ","log":"version[7.8.0], pid[10], build[oss/rpm/757314695644ea9a1dc2fecd26d1a43856725e65/2020-06-14T19:35:50.234439Z], OS[Linux/3.10.0-1127.13.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.7/11.0.7+10-LTS]"}
elasticsearch                        | {"type":"log","host":"e3933e9fe1c3","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2020-11-05T15:07:04.772Z","logger":"o.e.n.Node","timezone":"UTC","marker":"[localhost] ","log":"JVM home [/usr/lib/jvm/java-11-openjdk-11.0.7.10-4.el7_8.x86_64]"}
elasticsearch                        | {"type":"log","host":"e3933e9fe1c3","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2020-11-05T15:07:04.772Z","logger":"o.e.n.Node","timezone":"UTC","marker":"[localhost] ","log":"JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-1576227752267946562, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=rpm, -Des.bundled_jdk=true]"}
      | {"type":"log","host":"e3933e9fe1c3","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2020-11-05T15:07:05.821Z","logger":"o.e.e.NodeEnvironment","timezone":"UTC","marker":"[localhost] ","log":"heap size [1gb], compressed ordinary object pointers [true]"}
elasticsearch                        | {"type":"log","host":"e3933e9fe1c3","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2020-11-05T15:07:05.864Z","logger":"o.e.n.Node","timezone":"UTC","marker":"[localhost] ","log":"node name [localhost], node ID [cQDQte1HS-uxkHHIC9nUsQ], cluster name [elasticsearch-default]"}
elasticsearch                        | {"type":"log","host":"e3933e9fe1c3","level":"ERROR","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2020-11-05T15:07:08.103Z","logger":"o.e.b.ElasticsearchUncaughtExceptionHandler","timezone":"UTC","marker":"[localhost] ","log":"uncaught exception in thread [main]"}
elasticsearch                        | org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [xpack.security.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
elasticsearch                        |  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        | Caused by: java.lang.IllegalArgumentException: unknown setting [xpack.security.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
elasticsearch                        |  at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.node.Node.<init>(Node.java:370) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
elasticsearch                        |  ... 6 more
elasticsearch                        | uncaught exception in thread [main]
elasticsearch                        | java.lang.IllegalArgumentException: unknown setting [xpack.security.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
elasticsearch                        |  at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544)
elasticsearch                        |  at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489)
elasticsearch                        |  at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460)
elasticsearch                        |  at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431)
elasticsearch                        |  at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149)
elasticsearch                        |  at org.elasticsearch.node.Node.<init>(Node.java:370)
elasticsearch                        |  at org.elasticsearch.node.Node.<init>(Node.java:266)
elasticsearch                        |  at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227)
elasticsearch                        |  at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227)
elasticsearch                        |  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
elasticsearch                        |  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
elasticsearch                        |  at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
elasticsearch                        |  at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
elasticsearch                        |  at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
elasticsearch                        |  at org.elasticsearch.cli.Command.main(Command.java:90)
elasticsearch                        |  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
elasticsearch                        |  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
elasticsearch                        | For complete error details, refer to the log at /data/log/elasticsearch-default.log
elasticsearch exited with code 1

How can "[xpack.security.enabled]" be unknown setting when it is something actually mentioned in docs?
Additionaly I've already checked different versions of elasticsearch and the output is the same.

Here is how I'm setting that with docker compose ( docker-compose.yml):

---
version: '3'
services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:$ELASTIC_VERSION
    environment:
      - bootstrap.memory_lock=true
      - discovery.type=single-node
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=$ELASTIC_SECURITY
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200
    networks: ['stack']

  kibana:
    image: docker.elastic.co/kibana/kibana:$ELASTIC_VERSION
    environment:
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=$ELASTIC_PASSWORD
    ports: ['5601:5601']
    networks: ['stack']
    links: ['elasticsearch']
    depends_on: ['elasticsearch']

networks:
  stack: {}

.env file is:

ELASTIC_VERSION=7.9.3
ELASTIC_SECURITY=true
ELASTIC_PASSWORD=changeme

Thank you for the answer, this is in fact working. Somehow I didn't think of checking image repository.
Based on that I've noticed that I've been using company managed images and that's why xpack was not working. But right now I found out there are official and open source releases of elasticsearch so does it mean I can't use official image for work related project and there is no way to use built in login functionality?

You can freely use all the Basic-licensed features yourself and within your company,

Yeah, I understand that, but if I am not wrong it seems that xpack features are not available in Basic license hence no login feature and that's the problem.

This log snippet from above, version[7.8.0], pid[10], build[oss/rpm, seems to indicate that you are using an image based on the oss distribution and not the default one which includes x-pack.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.