Fleet setup - cannot enable kibana security

[bleed2002 (Patrick Kaiser)]

Hi there,

I want to try out central management for Elastic Agents, but i cannot get "Kibana security" enabled, although I'm sure it is configured correctly.

I'm running the stack on docker (19.03.12), ELK_VERSION is 7.11.1 .
Internal cluster communication is secured, for external access I'm using traefik as reverse proxy.

Currently I'm configuring the stack via host-mounted configuration files (elasticsearch.yml, kibana.yml), but I also tried to configure via environment variables with the same result.

Any ideas are highly appreciated to get this resolved

Here is my configuration:

1. docker-compose (left out services es02/es03):

version: '3.2'

services:
  es01:
    container_name: es01
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: $ELK_VERSION
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./elasticsearch/config/es01.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
      - /mnt/volume-elastic-data/es01-data:/usr/share/elasticsearch/data
      - es-certs:$CERTS_DIR
    environment:
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    healthcheck:
      test: curl --cacert $CERTS_DIR/ca/ca.crt -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5
    networks:
      - proxy_net
    labels:
      - 'traefik.enable=true'
      - 'traefik.port=9200'
      - 'traefik.protocol=https'
      - 'traefik.frontend.passHostHeader=true'
      - 'traefik.frontend.rule=Host:elastic.notarealdomain.com'

  kibana:
    container_name: kibana
    build:
      context: kibana/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
      - es-certs:$CERTS_DIR
    depends_on:
      - es01
    networks:
      - proxy_net
    environment:
      - "ELASTICSEARCH_USERNAME=kibana_system"
      - "ELASTICSEARCH_PASSWORD=$KIBANA_PASSWORD"
      - "ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=${CERTS_DIR}/ca/ca.crt"
    labels:
      - 'traefik.enable=true'
      - 'traefik.port=5601'
      - 'traefik.frontend.passHostHeader=true'
      - 'traefik.frontend.rule=Host:kibana.notarealdomain.com'

networks:
  proxy_net:
    external: true

volumes:
  es-certs:

2. elasticsearch.yml:

---
network.host: 0.0.0.0
cluster.name: "elk-cluster"
cluster.initial_master_nodes: es01,es02,es03
bootstrap.memory_lock: true

node.name: es01
discovery.seed_hosts: es02,es03

xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.license.self_generated.type: trial
xpack.monitoring.collection.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /usr/share/elasticsearch/config/certificates/es01/es01.key
xpack.security.http.ssl.certificate_authorities: /usr/share/elasticsearch/config/certificates/ca/ca.crt
xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certificates/es01/es01.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.certificate_authorities: /usr/share/elasticsearch/config/certificates/ca/ca.crt
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certificates/es01/es01.crt
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certificates/es01/es01.key

3. kibana.yml:

---
server.name: kibana.notarealdomain.com
server.host: 0.0.0.0

elasticsearch.hosts: [ "https://es01:9200", "https://es02:9200", "https://es03:9200" ]

xpack.reporting.kibanaServer.port: 443
xpack.reporting.kibanaServer.protocol: https
xpack.reporting.kibanaServer.hostname: kibana.notarealdomain.com
monitoring.ui.container.elasticsearch.enabled: true

xpack.security.enabled: true
xpack.encryptedSavedObjects.encryptionKey: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
xpack.fleet.enabled: true
xpack.fleet.agents.enabled: true
xpack.fleet.agents.elasticsearch.host: "https://elastic.notarealdomain.com:443"
xpack.fleet.agents.kibana.host: "https://kibana.notarealdomain.com:443"
xpack.fleet.agents.tlsCheckDisabled: false

Thanks, Patrick

[nchaulet (Nicolas Chaulet)]

Hi @bleed2002

If I understand correctly your config: Kibana is not served using an ssl connection, the ssl connection terminate on the traefik proxy.

If it's the case you should probably set the flag xpack.fleet.agents.tlsCheckDisabled: true to tell Fleet to not check for ssl as you are in a secure environment.

Let me know if it's help

[bleed2002 (Patrick Kaiser)]

Hi,
thanks for getting back to me and sorry for late answer, somehow i was not notified of your response.
No, the hole stack is using ssl internaly, but external communication is proxied through traefik.
All three elasticsearch and the kibana container share a common volume containing the certs, setup was done according to the docs here: Encrypting communications in an Elasticsearch Docker Container | Elasticsearch Reference [7.11] | Elastic
Of cource I could disable the ssl checks via the option you provided, but it feels wrong.
There should be clean/correct way, what do I miss here?

volumes:
  - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
  - es-certs:$CERTS_DIR

Thanks again, Patrick

[John_Schulz]

Hi @bleed2002,

Thanks for writing. We really appreciate your help.

Can you share more about the steps you took and what you observed? The original post mentioned wanting to try out central management for Elastic Agents. Were you in Kibana on a page like

Any URLs, screenshots, logs, or other info about the error message(s) are helpful in getting to the root of this and getting you moving again.

Thanks again

[bleed2002 (Patrick Kaiser)]

Hi @John_Schulz ,
thanks for getting back to me.
Yes, I see the following when I navigate to Management - Fleet - Agents in Kibana:

As you see, thre is an error for "Kibana Security" sugesting to set

xpack.security.enabled: true

what I did, please see my kibana.yml :

---
server.name: kibana.notarealdomain.com
server.host: 0.0.0.0

elasticsearch.hosts: [ "https://es01:9200", "https://es02:9200", "https://es03:9200" ]

xpack.reporting.kibanaServer.port: 443
xpack.reporting.kibanaServer.protocol: https
xpack.reporting.kibanaServer.hostname: kibana.notarealdomain.com
monitoring.ui.container.elasticsearch.enabled: true

xpack.security.enabled: true
xpack.encryptedSavedObjects.encryptionKey: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
xpack.fleet.enabled: true
xpack.fleet.agents.enabled: true
xpack.fleet.agents.elasticsearch.host: "https://elastic.notarealdomain.com:443"
xpack.fleet.agents.kibana.host: "https://kibana.notarealdomain.com:443"
xpack.fleet.agents.tlsCheckDisabled: false

I also tried to set this via an environment variable in my docker-compose file with the same result, setting is NOT effective.

...
kibana:
  container_name: kibana
  build:
    context: kibana/
    args:
      ELK_VERSION: $ELK_VERSION
  volumes:
    - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
    - es-certs:$CERTS_DIR
  depends_on:
    - es01
  networks:
    - proxy_net
  environment:
    - ELASTICSEARCH_USERNAME=kibana_system
    - ELASTICSEARCH_PASSWORD=$KIBANA_PASSWORD
    - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=${CERTS_DIR}/ca/ca.crt
    - xpack.security.enabled=true
  labels:
    - traefik.enable=true
    - traefik.port=5601
    - traefik.frontend.passHostHeader=true
    - traefik.frontend.rule=Host:kibana.notarealdomain.com
...

Maybe the problem has something to do with my reverse proxy setup? Internally the cluster communicates via the internal hostnames (es01,es02,es03) over https. Public access goes through the traefik reverse proxy (that also handles ssl termination via letsencrypt).

Thanks for your help!