Kibana UI does not work when kibana.yml file is mounted into docker container, but does work when config is set via env vars

[pred135]

I am having a very weird and frustrating issue with kibana. I am running the dockerhub elasticsearch and kibana containers on my kubernetes cluster. Here is the issue: I can run elasticsearch just fine, and also port forward port 9200 to a nodeport on my host machine and access elasticsearch via the browser, it prompts me to log in to elasticsearch, so I know it is working.

The issue is when i try to run kibana, i expose port 5601 and then create a nodeport to expose it on the host machine, but it just will not work, the browser just gives me a "ERR_CONNECTION_REFUSED". Now here is the weird part: when i run the kubernetes/docker container, i pass in the config options through environment variables (elastic host, username, password, etc.) and then it just works, i can access it through the browser no problem. But i would rather not put my config options in the environment variables, i would much rather use the kibana.yml file. So okay, i remove the env vars from my kubernetes file and add a file mount for the kibana.yml file and mount it to the container on path "/usr/share/kibana/config/kibana.yml". I can also exec into the container and see my file mounted at the correct location with the correct contents. But when i then try to access kibana through the browser i get the "ERR_CONNECTION_REFUSED" again. When i then check the kibana container logs there is absolutely nothing wrong at all, everything seems to be running. I then edited the kibana.yml file and added the line "logging.root.level: "trace"". I then restarted the container, and i know for sure that kibana is reading my mounted config file, because instead of just the info logs, it now shows all the logs (debug and trace). But then i go the the browser, and still no ui is loading. If i then remove the file mount from the kubernetes config and do it through the env vars again, the kibana ui works instantly again. The config between the vars and the kibana.yml file are exactly the same, no differences whatsoever. I don't understand the issue at all, there are also no differences in logs when i do it with env vars or with the config file mounted, but one will work in the browser, and the other won't for some reason.... I am running elasticsearch and kibana version 8.12.1 containers, anyone any idea?

[stephenb (Stephen Brown)]

Hi @pred135 Welcome to the community and thanks for the detailed description.

But I am a little confused are you running plain Docker? or Kubernetes?

Also you are going need to show the exact commands / mounts / config files so we can help... obviously

Here is a common issue Kibana only binding to localhost

so in your kibana.yml

set server.host: "0.0.0.0"

This is my entire kibana.yml (This ES has no security for this example)

# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
# elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.hosts: ["http://host.docker.internal:9200"]

This docker command works

docker run --name kib_test -v ./kibana.yml:/usr/share/kibana/config/kibana.yml -p 5601:5601 docker.elastic.co/kibana/kibana:8.11.3

Accessible from host browser at http://localhost:5601

[pred135]

omfg, that actually solved my issue, i spent all day and night yesterday trying to find the issue, i was convinced it was a file mount issue. Adding the server.host entry in the config file solved it. I still wonder why that line is 'implied' when only using env vars? because with those i wasnt setting that key either. But i have to explicitly set that one when using the kibana.yml file?

In any case, THANK YOU!!!

[stephenb (Stephen Brown)]

I'm a little confused about the difference too. One of those many little hidden mysteries of Docker It's just something I run into before