I've solved the errors for this issue that many others have had relative to not matching the host name and not verifying the first certificate by naturally adding the IP to the list of IPs generated in the instances.yml file as per the documentation (solves problem 1) and including the environment variable NODE_EXTRA_CA_CERTS=/usr/share/kibana/config/certs/ca/ca.crt to my docker composer (solves problem 2)
My issue is that Kibana still refuses to authenticate and configure my server but provides no error message (see title), just an empty string.
What follows is my docker compose code, minus the enterprise search container but that's moot for now since it depends on kibana which is not working to begin with:
setup:
container_name: elasticsearch-setup
image: docker.elastic.co/elasticsearch/elasticsearch:8.11.3
user: "0"
networks:
- elastic-network
volumes:
- elastic-certs:/usr/share/elasticsearch/config/certs
command: >
bash -c '
if [ ! -f config/certs/ca.zip ]; then
echo "Creating CA";
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f certs/certs.zip ]; then
echo "Creating certs";
echo -ne \
"instances:\n"\
" - name: es\n"\
" dns:\n"\
" - elasticsearch\n"\
" - localhost\n"\
" - ${DNSNAME}\n"\
" ip:\n"\
" - 127.0.0.1\n"\
" - ${IPADDRESS}\n"\
> config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
echo "Setting file permissions"
chown -R root:root config/certs;
find . -type d -exec chmod 750 \{\} \;;
find . -type f -exec chmod 640 \{\} \;;
echo "Waiting for Elasticsearch availability"
until curl -s --cacert config/certs/ca/ca.crt https://elasticsearch:9200 | grep -q "missing authentication credentials"; do sleep 10; done
echo "Elasticsearch is available"
echo "Setting kibana_system password"
until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTICPASS}" -H "Content-Type: application/json" https://elasticsearch:9200/_security/user/kibana_system/_password -d '{"password":"${KIBANAPASS}"}' | grep -q "200"; do sleep 10; done
echo "Password for kibana_system is set"
echo "All done!"
'
healthcheck:
test: ["CMD-SHELL", "[ -f config/certs/elasticsearch/elasticsearch.crt ]"]
interval: 1s
timeout: 5s
retries: 10
elasticsearch:
container_name: elasticsearch
depends_on:
setup:
condition: service_healthy
image: docker.elastic.co/elasticsearch/elasticsearch:8.11.3
user: "1000:0"
networks:
- elastic-network
volumes:
- elastic-certs:/usr/share/elasticsearch/config/certs
- elastic-search-data:/usr/share/elasticsearch/data
ports:
- "9200:9200"
environment:
- logger.discovery.level=debug
- node.name=elasticsearch
- cluster.name=${CLUSTERNAME}
- cluster.initial_master_nodes=elasticsearch
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.enrollment.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/elasticsearch/elasticsearch.key
- xpack.security.http.ssl.certificate=certs/elasticsearch/elasticsearch.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/elasticsearch/elasticsearch.key
- xpack.security.transport.ssl.certificate=certs/elasticsearch/elasticsearch.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=trial
- ES_JAVA_OPTS=-Xms16g -Xmx16g
- ELASTIC_PASSWORD=${ELASTICPASS}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test: ["CMD-SHELL", "curl -s --cacert config/certs/ca/ca.crt https://elasticsearch:9200 | grep -q 'missing authentication credentials'"]
interval: 10s
timeout: 10s
retries: 10
restart: unless-stopped
kibana:
container_name: kibana
depends_on:
elasticsearch:
condition: service_healthy
image: docker.elastic.co/kibana/kibana:8.11.3
user: "1000:0"
networks:
- elastic-network
volumes:
- elastic-certs:/usr/share/kibana/config/certs
- kibana-data:/usr/share/kibana/data
ports:
- "5601:5601"
environment:
- logging.root.level=trace
- server.name=kibana
- enterprisesearch.host=http://enterprisesearch:3002
- elasticsearch.hosts=https://elasticsearch:9200
- elasticsearch.username=kibana_system
- elasticsearch.password=${KIBANAPASS}
- elasticsearch.ssl.certificateAuthorities=/usr/share/kibana/config/certs/ca/ca.crt
- xpack.security.encryptionKey=${SECURITYKEY}
- xpack.encryptedSavedObjects.encryptionKey=${OBJECTSKEY}
- xpack.reporting.encryptionKey=${REPORTINGKEY}
- xpack.reporting.kibanaServer.hostname=kibana
- xpack.reporting.kibanaServer.protocol=https
- NODE_EXTRA_CA_CERTS=/usr/share/kibana/config/certs/ca/ca.crt
healthcheck:
test: ["CMD-SHELL", "curl -s -I --cacert config/certs/ca/ca.crt https://kibana:5601 | grep -q 'HTTP/1.1 302 Found'"]
interval: 10s
timeout: 10s
retries: 10
restart: unless-stopped
For reference: when i ran the default/test code for setting up a stack with enterprise search (with a simple copy paste and .env definition) from the elastic enterprise search 8.11 docker docs guide for setting it up, the stack worked fine and kibana connected.
This customisation is erroenous somewhere and with a blank error message im stuck without a clear way forward.
I should note that if I attempt to use curl and openssl directly from within the failed kibana container, it can connect and the ssl does verify. the consistent issue seems to be http requests that elastic is picking up when it expects https.
The openssl message says its likely http/0.9, and the elasticsearch container itself continuously logs that it receives a plaintext http request from kibana on an https port.
i have no yaml files or configuration files in my docker-compose build group. only a docker-compose.yml file and a .env file containing the interpolated variables