How to secure Elastic cluster with Docker + Traefik + SSL

[live2learn]

Hi there,

What I want to do
Set up an Elastic cluster (with Kibana) with Docker (via docker-compose). The cluster is secured (either username/password or API-Key) and lives behind a traefik reverse proxy which has LetsEncrypt enabled. From the same server (but also from other servers) I want to use Beats (Filebeats/Metricbeats) to send the logs to the cluster.

What I have done so far
Followed this guide: Running the Elastic Stack on Docker | Getting Started [7.15] | Elastic and did zero modifacation. The cluster was reachable and security was enabled (username/password). But since this is just an self signed certificate, Filebeat didn't trust that and gave me this error:

x509-certificate-signed-by-unknown-authority

Since I want to use Traefik as an reverse proxy, I tried to use the LetsEncrypt -Certificate and modified the elastic-docker-tls.yml to the following:

version: '2.2'

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    labels:
      - "traefik.enable=true"
      - "traefik.tcp.routers.elastic-tcp.tls=true"
      - "traefik.tcp.routers.elastic-tcp.entrypoints=https"
      - "traefik.tcp.routers.elastic-tcp.service=tcp-service"
      - "traefik.tcp.services.tcp-service.loadbalancer.server.port=9200"
      - "traefik.tcp.routers.elastic-tcp.rule=HostSNI(`es01.mydomain.tld`)"
      - "traefik.tcp.routers.elastic-tcp.tls.certresolver=le"
      - "traefik.tcp.routers.elastic-tcp.tls.options=myTLSOptions@file"
      - "traefik.tcp.routers.elastic-tcp.tls.passthrough=true"
    networks:
      - elastic
      - web

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic
  kib01:
    image: docker.elastic.co/kibana/kibana:${VERSION}
    container_name: kib01
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.kibana-sec.entrypoints=https"
      - "traefik.http.routers.kibana-sec.rule=Host(`kibana.mydomain.tld`)"
      - "traefik.http.routers.kibana-sec.tls=true"
      - "traefik.http.routers.kibana-sec.tls.certresolver=le"
      - "traefik.http.routers.kibana-sec.tls.options=myTLSOptions@file"
      - "traefik.http.services.kibana-sec.loadbalancer.server.port=5601"
      - "traefik.http.services.kibana-sec.loadbalancer.server.scheme=https"
    environment:
      SERVERNAME: localhost
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: MYPASSWORD
    networks:
      - elastic
      - web
volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge
  web:
    external: true

What does not work
Now the cluster won't start, because if xpack.security.enabled is set to true, I have to provide the key and/or the path to the certificate.

java.lang.IllegalArgumentException: a key must be provided to run as a server. the key should be configured using the [xpack.security.http.ssl.key] or [xpack.security.http.ssl.keystore.path] setting

The certificates are stored by Traefik in an acme.json-File and I don't know, how to provide it to the cluster.

Thank you in advance!

[TimV (Tim Vernum)]

This is because you have set

xpack.security.http.ssl.enabled=true

If you want to turn on SSL for HTTP in Elasticsearch, then you need to configure it - enabling SSL on a server (any server) requires a certificate and key.

You don't have to use the same certificate that Traefik uses, you can generate a new one with elasticsearch-certutil http

Or you can turn off SSL for HTTP.

[live2learn]

Hi Tim, thank you very much!

After turning off SSL for HTTP it worked, but then I couldn't use all of the functionality the elastic stack offers (couldn't activate rules/alerts as it requires an API key which depends on SSL).

Anyway, I figured it out now. Traefik has to communicate with its internal network via SSL as well.
To achieve this, I had to add this label:

- "traefik.http.services.kibana-sec.loadbalancer.server.scheme=https"

Since it is a self signed certificate, I also had to add this to my traefik.yml, to ignore this in the internal network:

serversTransport:
   insecureSkipVerify: true

Here is my docker-compose file, for future seekers:
Note that, I followed the guide as stated in my first post and just added traefik labels and traefik.yml. You will have to create certs and generate the credentials.

version: '2.2'

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.authc.api_key.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.elastic.entrypoints=http"
      - "traefik.http.routers.elastic.rule=Host(`mydomain.tld.de`)"
      - "traefik.http.routers.elastic.middlewares=https-redirect@file"
      - "traefik.http.routers.elastic-sec.entrypoints=https"
      - "traefik.http.routers.elastic-sec.rule=Host(`mydomain.tld`)"
      - "traefik.http.routers.elastic-sec.tls=true"
      - "traefik.http.routers.elastic-sec.tls.certresolver=le"
      - "traefik.http.routers.elastic-sec.tls.options=myTLSOptions@file"
      - "traefik.http.services.elastic-sec.loadbalancer.server.port=9200"
      - "traefik.http.services.elastic-sec.loadbalancer.server.scheme=https"
    networks:
      - elastic
      - web

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es02/es02.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es02/es02.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es02/es02.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es02/es02.key
    ulimits:
      memlock:
        soft: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es03/es03.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es03/es03.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es03/es03.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es03/es03.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    networks:
      - elastic
  kib01:
    image: docker.elastic.co/kibana/kibana:${VERSION}
    container_name: kib01
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.kibana.entrypoints=http"
      - "traefik.http.routers.kibana.rule=Host(`mydomain.tld`)"
      - "traefik.http.routers.kibana.middlewares=https-redirect@file"
      - "traefik.http.routers.kibana-sec.entrypoints=https"
      - "traefik.http.routers.kibana-sec.rule=Host(`mydomain.tld`)"
      - "traefik.http.routers.kibana-sec.tls=true"
      - "traefik.http.routers.kibana-sec.tls.certresolver=le"
      - "traefik.http.routers.kibana-sec.tls.options=myTLSOptions@file"
      - "traefik.http.services.kibana-sec.loadbalancer.server.port=5601"
      - "traefik.http.services.kibana-sec.loadbalancer.server.scheme=https"
    environment:
      SERVERNAME: localhost
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: CHANGEME
      XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: Z6K7XnTndRoPbBRT3adzVgfdmukfLx5FT34sdg3w
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: $CERTS_DIR/kib01/kib01.key
      SERVER_SSL_CERTIFICATE: $CERTS_DIR/kib01/kib01.crt
    volumes:
      - certs:$CERTS_DIR
    networks:
      - elastic
      - web
volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local
  certs:
    driver: local

networks:
  elastic:
    driver: bridge
  web:
    external: true