Docker container exiting with code 0 & no logs

Elastic Stack Kibana
1

I have elasticsearch up and running in a Docker container and accessible from other containers with http://elasticsearch. I'm trying to connect up Kibana and no matter what I try, the container simply exits with error code 0 and with no logs whatsoever (i.e. when running docker logs [id])

This is the command I'm using:

docker run --network private \
  -e ELASTICSEARCH_URL=http://elasticsearch:9200 \
  -e XPACK_MONITORING_ENABLED=false \
  -p 5601:5601 \
  docker.elastic.co/kibana/kibana:6.0.0

I've tried both the kibana and kibana-oss versions. I've tried diabling x-pack, I've tried turning on verbose logging. Always with the same results.

I've also tried attaching onto /bin/bash within the container and running the kibana start command directly. Again, nothing.

Either I've missed something massively obvious or I'm going insane... I've searched in this forum, in GitHub issues and on Google/StackOverflow with no similar scenarios found..

Please help!

Thanks.

2

Is the Kibana container able to resolve "http://elasticsearch"? If not, it won't be able to connect to the Elasticsearch container, and won't start up. If that's the case though, you should see a message about that in the logs, which brings me to my next question; what do you see in the Kibana logs?

3

Hi Joe,

Thanks for your response. The main problem seems to be there are no logs! No matter what I do, nothing is printed to stdout/err in the container.

Interestingly, the plot thickens...

$ docker run -it --network private \
>   node:6.9.1 \
>   /bin/bash -c "ping elasticsearch"
PING elasticsearch (10.0.1.11): 56 data bytes
64 bytes from 10.0.1.11: icmp_seq=0 ttl=64 time=0.045 ms
64 bytes from 10.0.1.11: icmp_seq=1 ttl=64 time=0.058 ms
64 bytes from 10.0.1.11: icmp_seq=2 ttl=64 time=0.071 ms
^C--- elasticsearch ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.045/0.064/0.079/0.000 ms

$ docker run -it --network private \
>   docker.elastic.co/kibana/kibana:6.0.0 \
>   /bin/bash -c "ping elasticsearch"
ping: elasticsearch: Name or service not known
4

Sounds like you could be running out of memory. If you are not running Docker on real Linux, but Windows or macOS instead, make sure you've bumped the memory in the VM as shown in: https://github.com/elastic/stack-docker#prerequisites

When there are no logs it helps to have docker events running while you run your kibana container to see what's happening in the background.

5

Thanks for your response Dimitrios. How much memory does Kibana require? I've tried it on a few different VMs, e.g.:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        3.3G        581M        409M        3.8G        3.4G

From docker events, I get the following:

2017-11-29T07:47:43.520676049-05:00 container create 9b42898a05468b58af5e2befd1a92d0993e275d22e7e07332a6297080d2bf33a (build-date=20170911, image=docker.elastic.co/kibana/kibana:6.0.0, license=GPLv2, maintainer=Elastic Docker Team <docker@elastic.co>, name=pedantic_blackwell, vendor=CentOS)
2017-11-29T07:47:43.524123293-05:00 container attach 9b42898a05468b58af5e2befd1a92d0993e275d22e7e07332a6297080d2bf33a (build-date=20170911, image=docker.elastic.co/kibana/kibana:6.0.0, license=GPLv2, maintainer=Elastic Docker Team <docker@elastic.co>, name=pedantic_blackwell, vendor=CentOS)
2017-11-29T07:47:43.963930053-05:00 network connect dirxm58uuhtosk0q6rhhzhn14 (container=9b42898a05468b58af5e2befd1a92d0993e275d22e7e07332a6297080d2bf33a, name=private, type=overlay)
2017-11-29T07:47:45.360665124-05:00 container start 9b42898a05468b58af5e2befd1a92d0993e275d22e7e07332a6297080d2bf33a (build-date=20170911, image=docker.elastic.co/kibana/kibana:6.0.0, license=GPLv2, maintainer=Elastic Docker Team <docker@elastic.co>, name=pedantic_blackwell, vendor=CentOS)
2017-11-29T07:47:45.945409716-05:00 container die 9b42898a05468b58af5e2befd1a92d0993e275d22e7e07332a6297080d2bf33a (build-date=20170911, exitCode=0, image=docker.elastic.co/kibana/kibana:6.0.0, license=GPLv2, maintainer=Elastic Docker Team <docker@elastic.co>, name=pedantic_blackwell, vendor=CentOS)
2017-11-29T07:47:46.179614882-05:00 network disconnect dirxm58uuhtosk0q6rhhzhn14 (container=9b42898a05468b58af5e2befd1a92d0993e275d22e7e07332a6297080d2bf33a, name=private, type=overlay)

No obvious clues :confused:

Oh and I forgot to add - I'm running it on 'real' Linux:

$ cat /etc/*release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.3 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.3"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.3 (Maipo)"
6

A few things to try:

  1. Can you provide the docker info output?
  2. How's your free disk space?
  3. Do you think you can wipe things clean with docker system prune -a first?
  4. To help reproduction can you use this Vagrantfile and then:
vagrant up
vagrant ssh
docker network create private
docker run -d --name elasticsearch --network private -p 9200:9200 docker.elastic.co/elasticsearch/elasticsearch:6.0.0
docker run --network private -p 5601:5601 docker.elastic.co/kibana/kibana:6.0.0

The procedure above works fine for me in the clean Vagrant env and will help us isolate if there's a host related issue or something else.

7

Also noticed the output from docker info that you pasted in the GH issue.

Possibly not related to the issue you are reporting, but the output from docker info provides some warning that would certainly cause issues in the future:

WARNING: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior.
         Reformat the filesystem with ftype=1 to enable d_type support.
         Running without d_type support will not be supported in future releases.
WARNING: bridge-nf-call-ip6tables is disabled
8

Thanks for clarifying, unfortunately I am constrained by my environment at the moment and am unable to use your Vagrant script. Thank you for providing it though. The warnings will be addressed in the future.

I've done a little more digging and it seems to fall down to the node installation. Using this Dockerfile works fine. But if I switch the version of node to the same as packaged with Kibana (v6.11.5) I get exactly the same symptoms.. (quits with exit code 0)

For now I shall just use my own image as I don't have any more time to debug unfortunately.

Thank you for your help though, and maybe this will help someone else in the future..

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.