Connection Refused Docker Kibana

Elastic Stack Kibana
1

Kibana connection refused,
(1) Plain vanilla linux ubuntu 17. Docker 1.12.6 (I've got similar behavior on centos 7 / docker 17)

(2.1) using docker elasticsearch 5.6.1. This command to start es

sudo docker run -d -p 9200:9200 -p 9300:9300 \
    -e "discovery.type=single-node" \
    docker.elastic.co/elasticsearch/elasticsearch:5.6.1

(2.2) can curl -v -u elastic:changeme localhost:9200
(2.3) can curl -v -u elastic:changeme 192.168.43.93:9200

(3) This is my kibana.yml

server.host: 192.168.43.93
server.name: scotts-kibana
elasticsearch.url: http://192.168.43.93:9200
elasticsearch.username: elastic
elasticsearch.password: changeme
xpack.monitoring.ui.container.elasticsearch.enabled: false
xpack.monitoring.enabled: false

(4.1) using docker kibana 5.6.1 This command to start kibana bash

sudo docker run -it -p 5601:5601 \
    -v /home/scott/kibana.yml:/usr/share/kibana/config/kibana.yml \
    docker.elastic.co/kibana/kibana:5.6.1 \
    bash

(4.2) can curl -v -u elastic:changeme 192.168.43.93:9200
(4.3) kill that docker container

(5.1) using docker kibana 5.6.1 This command to start kibana

sudo docker run -it -p 5601:5601 \
    -v /home/scott/kibana.yml:/usr/share/kibana/config/kibana.yml \
    docker.elastic.co/kibana/kibana:5.6.1

(5.2) Fails curl -u elastic:changeme 192.168.43.93:5601

Failed to connect to 192.168.43.93 port 5601: Connection refused
2

Inside each container, what does sudo netstat -nltp tell you? Are the ports you expect to be up up?

3

I'm starting to think it is related to the xpack security. If I enable it by commenting out the two xpack settings that enabled: false, the root curl still refuses which was the misdirection (didn't curl -v) but it shows a redirect to /login.

4

That's very weird. If possible, definitely log it with GitHub issues.

5

So I'm moving away from the xpack idea. It's enabled (all default settings) but I still have the network looking issue.

sudo netstat -nltp doesn't work for me, I tried it inside docker exec -it kibana bash

bash-4.2$ netstat
bash: netstat: command not found

Regardless, when I am inside the container I can successfully curl localhost:5601/app/kibana

It's outside the container that isn't working. I don't know where the problem lies (obviously), is it in the kibana image? Is it Docker? Is it my networking configuration of the box. I mean the same exact box works for elasticsearch. Different port though, but -p 9200:5601 for example didn't work either.

It does behave differently based on how I start up the container. if I do -p 5601:5601 neither of these curls work:

curl localhost:5601
curl 127.0.0.1:5601

but if I do --network=host both curls work.

Again, same box does one thing with es another with kibana. Docker is the same in both cases, this leads me to look at the image. I know for elasticsearch, I had to configure the bind and publish hosts. Can you do this for Kibana? It didn't complain but it didn't help.

network.bind_host: 0.0.0.0
network.publish_host: 123.45.67.88

My gut tells me it's something with the image and the way it is configured, but I can't prove that. Again, docker and the box/os are the same in es / kibana

6

I've run both ElasticSearch and Kibana 5.6.1 images on the same exact aws instance, at the same time, running

  • CentOS Linux release 7.3.1611 (Core)
  • Docker version 17.06.1-ce, build 874a737
  • It has an ip setup by a network interface (eni, 10.207.67.250)
  • It has a Route53 asscoiated with that eni (kibana.mycompany.com)

Also I've also run both on Ubuntu 17.04. It doesn't have the route53, but it has localhost, 127 and an ip. It succeeds and fails identically to the centos setup

  • Ubuntu 17.04
  • Docker version 1.12.6, build 78d1802
Legend:
  * worked
  x failed

ElasticSearch with container networking and port mapping

I can get to the service in all cases.

docker run -d \
    -p 9200:9200 -p 9300:9300 \
    -e "discovery.type=single-node" \
    docker.elastic.co/elasticsearch/elasticsearch:5.6.1 

* curl -v -u elastic:changeme localhost:9200/_cat/health
* curl -v -u elastic:changeme 127.0.0.1:9200/_cat/health
* curl -v -u elastic:changeme 10.207.67.250:9200/_cat/health
* curl -v -u elastic:changeme kibana.mycompany.com:9200/_cat/health

ElasticSearch with host networking

I can get to the service in all cases.

docker run -d \
    --network=host \
    -e "discovery.type=single-node" \
    docker.elastic.co/elasticsearch/elasticsearch:5.6.1

* curl -v -u elastic:changeme localhost:9200/_cat/health
* curl -v -u elastic:changeme 127.0.0.1:9200/_cat/health
* curl -v -u elastic:changeme 10.207.67.250:9200/_cat/health
* curl -v -u elastic:changeme kibana.mycompany.com:9200/_cat/health

Kibana.yml

server.name: kibana-node-1
elasticsearch.url: http://kibana.mycompany.com:9200
elasticsearch.username: elastic
elasticsearch.password: changeme
xpack.monitoring.ui.container.elasticsearch.enabled: true

Kibana with container networking and port mapping

I cannot get to the service at all.

docker run -d \
    -p 5601:5601 
    -v /home/centos/kibana.yml:/usr/share/kibana/config/kibana.yml \
    docker.elastic.co/kibana/kibana:5.6.1

x curl -v localhost:5601/app/kibana
x curl -v 127.0.0.1:5601/app/kibana
x curl -v 10.207.67.250:5601/app/kibana
x curl -v kibana.mycompany.com:5601/app/kibana

Kibana with host networking

I can get to the service locally.

docker run -d \
    --network=host \
    -v /home/centos/kibana.yml:/usr/share/kibana/config/kibana.yml \
    docker.elastic.co/kibana/kibana:5.6.1
    
* curl -v localhost:5601/app/kibana
* curl -v 127.0.0.1:5601/app/kibana
x curl -v 10.207.67.250:5601/app/kibana
x curl -v kibana.mycompany.com:5601/app/kibana
7

what do these configuration settings do (from https://www.elastic.co/guide/en/kibana/current/settings.html)

server.host
elasticsearch.preserveHost

Since this is running in a docker container, does the default server.host of "localhost" cause problems? Should this maybe be set to the ip address of the host machine?

For my actual elasticsearch nodes, I had set these up in elasticsearch.yml. Is the server.host the equivalent? Just trying to work through this.

network.bind_host: 0.0.0.0
network.publish_host: 10.207.67.250
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.