Hi there,
What I want to do
Set up an Elastic cluster (with Kibana) with Docker (via docker-compose). The cluster is secured (either username/password or API-Key) and lives behind a traefik reverse proxy which has LetsEncrypt enabled. From the same server (but also from other servers) I want to use Beats (Filebeats/Metricbeats) to send the logs to the cluster.
What I have done so far
Followed this guide: Running the Elastic Stack on Docker | Getting Started [7.15] | Elastic and did zero modifacation. The cluster was reachable and security was enabled (username/password). But since this is just an self signed certificate, Filebeat didn't trust that and gave me this error:
x509-certificate-signed-by-unknown-authority
Since I want to use Traefik as an reverse proxy, I tried to use the LetsEncrypt -Certificate and modified the elastic-docker-tls.yml to the following:
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.license.self_generated.type=trial
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
labels:
- "traefik.enable=true"
- "traefik.tcp.routers.elastic-tcp.tls=true"
- "traefik.tcp.routers.elastic-tcp.entrypoints=https"
- "traefik.tcp.routers.elastic-tcp.service=tcp-service"
- "traefik.tcp.services.tcp-service.loadbalancer.server.port=9200"
- "traefik.tcp.routers.elastic-tcp.rule=HostSNI(`es01.mydomain.tld`)"
- "traefik.tcp.routers.elastic-tcp.tls.certresolver=le"
- "traefik.tcp.routers.elastic-tcp.tls.options=myTLSOptions@file"
- "traefik.tcp.routers.elastic-tcp.tls.passthrough=true"
networks:
- elastic
- web
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.license.self_generated.type=trial
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data02:/usr/share/elasticsearch/data
networks:
- elastic
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.license.self_generated.type=trial
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data03:/usr/share/elasticsearch/data
networks:
- elastic
kib01:
image: docker.elastic.co/kibana/kibana:${VERSION}
container_name: kib01
labels:
- "traefik.enable=true"
- "traefik.http.routers.kibana-sec.entrypoints=https"
- "traefik.http.routers.kibana-sec.rule=Host(`kibana.mydomain.tld`)"
- "traefik.http.routers.kibana-sec.tls=true"
- "traefik.http.routers.kibana-sec.tls.certresolver=le"
- "traefik.http.routers.kibana-sec.tls.options=myTLSOptions@file"
- "traefik.http.services.kibana-sec.loadbalancer.server.port=5601"
- "traefik.http.services.kibana-sec.loadbalancer.server.scheme=https"
environment:
SERVERNAME: localhost
ELASTICSEARCH_URL: https://es01:9200
ELASTICSEARCH_HOSTS: https://es01:9200
ELASTICSEARCH_USERNAME: kibana_system
ELASTICSEARCH_PASSWORD: MYPASSWORD
networks:
- elastic
- web
volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local
networks:
elastic:
driver: bridge
web:
external: true
What does not work
Now the cluster won't start, because if xpack.security.enabled is set to true, I have to provide the key and/or the path to the certificate.
java.lang.IllegalArgumentException: a key must be provided to run as a server. the key should be configured using the [xpack.security.http.ssl.key] or [xpack.security.http.ssl.keystore.path] setting
The certificates are stored by Traefik in an acme.json-File and I don't know, how to provide it to the cluster.
Thank you in advance!
TimV
(Tim Vernum)
November 23, 2021, 10:20am
2
This is because you have set
xpack.security.http.ssl.enabled=true
If you want to turn on SSL for HTTP in Elasticsearch, then you need to configure it - enabling SSL on a server (any server) requires a certificate and key.
You don't have to use the same certificate that Traefik uses, you can generate a new one with elasticsearch-certutil http
Or you can turn off SSL for HTTP.
1 Like
Hi Tim, thank you very much!
After turning off SSL for HTTP it worked, but then I couldn't use all of the functionality the elastic stack offers (couldn't activate rules/alerts as it requires an API key which depends on SSL).
Anyway, I figured it out now. Traefik has to communicate with its internal network via SSL as well.
To achieve this, I had to add this label:
- "traefik.http.services.kibana-sec.loadbalancer.server.scheme=https"
Since it is a self signed certificate, I also had to add this to my traefik.yml, to ignore this in the internal network:
serversTransport:
insecureSkipVerify: true
Here is my docker-compose file, for future seekers:
Note that, I followed the guide as stated in my first post and just added traefik labels and traefik.yml. You will have to create certs and generate the credentials.
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.license.self_generated.type=trial
- xpack.security.enabled=true
- xpack.security.authc.api_key.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
- xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
- xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
- certs:$CERTS_DIR
labels:
- "traefik.enable=true"
- "traefik.http.routers.elastic.entrypoints=http"
- "traefik.http.routers.elastic.rule=Host(`mydomain.tld.de`)"
- "traefik.http.routers.elastic.middlewares=https-redirect@file"
- "traefik.http.routers.elastic-sec.entrypoints=https"
- "traefik.http.routers.elastic-sec.rule=Host(`mydomain.tld`)"
- "traefik.http.routers.elastic-sec.tls=true"
- "traefik.http.routers.elastic-sec.tls.certresolver=le"
- "traefik.http.routers.elastic-sec.tls.options=myTLSOptions@file"
- "traefik.http.services.elastic-sec.loadbalancer.server.port=9200"
- "traefik.http.services.elastic-sec.loadbalancer.server.scheme=https"
networks:
- elastic
- web
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.license.self_generated.type=trial
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=$CERTS_DIR/es02/es02.key
- xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.http.ssl.certificate=$CERTS_DIR/es02/es02.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.transport.ssl.certificate=$CERTS_DIR/es02/es02.crt
- xpack.security.transport.ssl.key=$CERTS_DIR/es02/es02.key
ulimits:
memlock:
soft: -1
volumes:
- data02:/usr/share/elasticsearch/data
- certs:$CERTS_DIR
networks:
- elastic
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.license.self_generated.type=trial
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=$CERTS_DIR/es03/es03.key
- xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.http.ssl.certificate=$CERTS_DIR/es03/es03.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.transport.ssl.certificate=$CERTS_DIR/es03/es03.crt
- xpack.security.transport.ssl.key=$CERTS_DIR/es03/es03.key
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data03:/usr/share/elasticsearch/data
- certs:$CERTS_DIR
networks:
- elastic
kib01:
image: docker.elastic.co/kibana/kibana:${VERSION}
container_name: kib01
labels:
- "traefik.enable=true"
- "traefik.http.routers.kibana.entrypoints=http"
- "traefik.http.routers.kibana.rule=Host(`mydomain.tld`)"
- "traefik.http.routers.kibana.middlewares=https-redirect@file"
- "traefik.http.routers.kibana-sec.entrypoints=https"
- "traefik.http.routers.kibana-sec.rule=Host(`mydomain.tld`)"
- "traefik.http.routers.kibana-sec.tls=true"
- "traefik.http.routers.kibana-sec.tls.certresolver=le"
- "traefik.http.routers.kibana-sec.tls.options=myTLSOptions@file"
- "traefik.http.services.kibana-sec.loadbalancer.server.port=5601"
- "traefik.http.services.kibana-sec.loadbalancer.server.scheme=https"
environment:
SERVERNAME: localhost
ELASTICSEARCH_URL: https://es01:9200
ELASTICSEARCH_HOSTS: https://es01:9200
ELASTICSEARCH_USERNAME: kibana_system
ELASTICSEARCH_PASSWORD: CHANGEME
XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: Z6K7XnTndRoPbBRT3adzVgfdmukfLx5FT34sdg3w
ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
SERVER_SSL_ENABLED: "true"
SERVER_SSL_KEY: $CERTS_DIR/kib01/kib01.key
SERVER_SSL_CERTIFICATE: $CERTS_DIR/kib01/kib01.crt
volumes:
- certs:$CERTS_DIR
networks:
- elastic
- web
volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local
certs:
driver: local
networks:
elastic:
driver: bridge
web:
external: true
system
(system)
Closed
December 27, 2021, 6:32pm
4
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.