Not able to connect to Kibana from any IP other than localhost loopback address

Elastic Stack Kibana
1

Hello, I am able to connect to http://localhost:5601 on my ubuntu 18.04 machine that is running docker and the ELK stack, but not able to connect to my Ethernet IP.

mneely@umbrella-kibana:~$ curl -i http://localhost:5601
HTTP/1.1 302 Found
location: /app/kibana
kbn-name: kibana
kbn-xpack-sig: 6ac3af7a9155783217983f0fb9fc6b6b
content-type: text/html; charset=utf-8
cache-control: no-cache
content-length: 0
connection: close
Date: Wed, 08 Apr 2020 20:28:23 GMT

mneely@umbrella-kibana:~$ curl -i http://172.18.3.253:5601
curl: (7) Failed to connect to 172.18.3.253 port 5601: Connection refused
mneely@umbrella-kibana:~$

mneely@umbrella-kibana:~$ ifconfig
> docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
> inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
> ether 02:42:bb:9e:ff:dd txqueuelen 0 (Ethernet)
> RX packets 0 bytes 0 (0.0 B)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 0 bytes 0 (0.0 B)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet 172.18.3.253 netmask 255.255.252.0 broadcast 172.18.3.255
> inet6 fe80::250:56ff:feb2:d86b prefixlen 64 scopeid 0x20
> ether 00:50:56:b2:d8:6b txqueuelen 1000 (Ethernet)
> RX packets 64435 bytes 90229013 (90.2 MB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 21423 bytes 1712637 (1.7 MB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
> inet 10.1.66.0 netmask 255.255.255.255 broadcast 0.0.0.0
> inet6 fe80::c822:9ff:fe93:90b prefixlen 64 scopeid 0x20
> ether ca:22:09:93:09:0b txqueuelen 0 (Ethernet)
> RX packets 0 bytes 0 (0.0 B)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 0 bytes 0 (0.0 B)
> TX errors 0 dropped 14 overruns 0 carrier 0 collisions 0
>
> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
> inet 127.0.0.1 netmask 255.0.0.0'

---
## Default Kibana configuration from kibana-docker.
## from https://github.com/elastic/kibana-docker/blob/master/build/kibana/config/kibana.yml
#
server.name: kibana
server.host: "0.0.0.0"
elasticsearch.url: http://localhost:9200
server.port: 5601
logging.dest: /var/log/kibana/kibana.log

mneely@umbrella-kibana:~$ sudo ufw status
Status: inactive

Netstat shows only listening on port 5601, so I imagine that is the problem but I have no idea what is missing to allow me to bind to my ethernet interface.

mneely@umbrella-kibana:~$ netstat -anp | grep 5601
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:5601 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:57530 127.0.0.1:5601 TIME_WAIT -

2

Modify the network.host setting in kibana.yml could change the listen address.

[Update] This is for Elasticsearch; check the next comment below for Kibana's

3

No such luck.

My understanding is that the network.host settings does the trick for elasticsearch (which I do have it set to 0.0.0.0) but not a factor in kibana.yml file.

4

Oh my mistake

It should be server.host setting in Kibana.
You can search for it in Configuring Kibana

5

Yes, that's set. See my kibana.yml in the 4th text block in my original question.

6

Can you paste the command you used for running the containers?

If the kibana instance runs in a container in bridge mode, it wouldn't be able to connect to Elasticsearch through http://localhost:9200

7

I am not sure what you mean by the command to run the containers?

I am able to connect to the elasticsearch itself from the VM I am running the ELK stack on.

mneely@umbrella-kibana:~$ curl -i http://localhost:9200
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 493

{
"name" : "IybWIcl",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "WsLLXk1WQs2a2T0-qWAMRg",
"version" : {
"number" : "6.8.8",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "2f4c224",
"build_date" : "2020-03-18T23:22:18.622755Z",
"build_snapshot" : false,
"lucene_version" : "7.7.2",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
mneely@umbrella-kibana:~$

A little about my setup - I'm running this on ubuntu 18.04 with docker, and the ELK stack installed.

Here is my network config -

mneely@umbrella-kibana:~$ ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:34:85:17:45 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.18.3.253 netmask 255.255.252.0 broadcast 172.18.3.255
inet6 fe80::250:56ff:feb2:d86b prefixlen 64 scopeid 0x20
ether 00:50:56:b2:d8:6b txqueuelen 1000 (Ethernet)
RX packets 4714 bytes 476766 (476.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 713 bytes 93528 (93.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.1.66.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::485f:82ff:feb1:5b1 prefixlen 64 scopeid 0x20
ether 4a:5f:82:b1:05:b1 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 15 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 438791 bytes 80577427 (80.5 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 438791 bytes 80577427 (80.5 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

8

Did you installed Elasticsearch and Kibana as a system service (e.g. apt-get install kibana)?
If yes, could you make sure that the configuration file is the one you modified? Use the command systemctl cat kibana to show your kibana parameters, mine has ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"

Also, could you search for the string "message":"http server running at http:// in the kibana log file /var/log/kibana/kibana.log? Mine which listens to 0.0.0.0 will have the message "message":"http server running at http://0.0.0.0:5601" logged

9

Wow, that's it.

ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"

And my logs never worked because I have edited the yml file in my local directory to setup logging versus the above. Do you recommend changing the kibana parameters to point to my local yml file or to just edit the yml files the apt install created?

10

Usually we will backup the original configuration file to something like /etc/kibana/kibana.yml.bak and just edit the file /etc/kibana/kibana.yml, so that when you screw up the configuration file, you can have something to revert with.

11

This got it working. Thank you for your help!

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.