Unable to access external ElasticSearch Endpoint from within Kibana Docker Container

Elastic Stack Kibana
1

I'm trying to access an external elasticsearch URL from within my Kibana Docker Container running on an AWS EC2 Instance.

Though I'm able to CURL from the EC2 Instance - the same CURL doesn't work from within the Docker container. Naturally, since my EC2 Instance can connect to the elasticsearch URL, its not a SecurityGroup/Firewall issue.
Essentially this CURL works in the EC2 Instance but NOT within the Docker Container. Why would that be? Consequently, kibana fails to come up as indicated below with the error of timeout. Interesting is that its not a Connection Refused but a timeout.

curl https://elasticsearch.myorg.com

Dockerfile:

FROM docker.elastic.co/kibana/kibana:6.5.0
ENV ELASTICSEARCH_URL=https://elasticsearch.myorg.com
EXPOSE 5601
CMD ["/usr/local/bin/kibana-docker"]

This is the error I'm getting from within the docker container logs:

{"type":"log","@timestamp":"2018-11-14T16:58:28Z","tags":["status","plugin:logstash@6.4.2","error"],"pid":1,"state":"red","message":"Status changed from red to red - Request Timeout after 30000ms","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2018-11-14T16:59:28Z","tags":["license","warning","xpack"],"pid":1,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 30000ms"}
2

What happens if you run:

docker run --rm byrnedo/alpine-curl https://elasticsearch.myorg.com

?

3

And if you explicitly run:

docker run --rm docker.elastic.co/kibana/kibana:6.5.0 curl https://elasticsearch.myorg.com

Does that fail?

What about with your custom image?

4

Actually, the logs from your custom image contain references to plugin:logstash@6.4.2, but your Dockerfile is based on Kibana 6.5.0. Did a step fail between rebuilding the custom image and running it?

5

So i managed to get this to "work" but using the following command. Found some ancient post in stack overflow where someone wasn't exactly facing this but the options seemed to be a non-binding loopback address call that magically started working:

docker run --net=host -d -p 5601:5601 my-kibana-image:1

But the "non-binding loopback" is just jargon to me. If anyone can explain WHY it worked - I would consider the matter truly resolved (the magic behind --net=host)

1 Like
6

"non-binding loopback" doesn't mean much to me either.

However, --net=host connects the container process (Kibana) directly the network interface(s) on the host system, rather than running it in its own network namespace with its own virtual interfaces.

However, we haven't discovered why this is necessary in your environment. You should not have to use --net=host to allow Kibana to "see out", like this:

$ docker run --rm docker.elastic.co/kibana/kibana:6.5.0 curl -s http://example.org | head -n4
<!doctype html>
<html>
<head>
    <title>Example Domain</title>
7

our EC2 Instances sit behind Bastion Hosts. Could that have something to do with this?
It would be strange since our "regular" non-docker based kibana run on the same bastion Ec2s without problem

8

It's hard to say. My first guess would be something about the routing rules or netfilter rules on the Docker host. Docker networking works because of a bunch of subtle stuff in netfilter. Just have a look at the output of iptables -L -v to see how much trickery is going on! :slight_smile:

Personally, I would start poking around with tcpdump to see how far the packets are making it. For example, do they ever leave the Docker host and actually get on the wire?

9

Do proxies matter from within Docker Containers?
Our EC2 based kibana drive off our proxies and work fine. I did try to set the proxies using ENV from within the container but that didnt do anyhting

10

Are you saying that you access Kibana through a proxy, or that Kibana accesses Elasticsearch through a proxy?

Even if it's the latter, I wouldn't expect that to be worked around by --net=host. If Kibana needs a proxy to contact Elasticsearch, and it's not correctly configured, it won't matter which network interface it's connected to.

What happens if you run:

docker run --rm byrnedo/alpine-curl http://example.org

?

11

Had to repackage alpine-curl since i could not access the public repo!

Running curl to the elasticsearch endpoint WITH --net=host works perfectly:

docker run --net=host artifactory.cloud.myhost.com/alpine-curl https:/my-elastic-endpoint

image
image972×411 12 KB

Running curl to the elasticsearch endpoint WITHOUT --net=host hangs forever:

docker run artifactory.cloud.myhost.com/alpine-curl https:/my-elastic-endpoint

image
image1563×42 3.9 KB

12

OK. That's really interesting. I'd say you definitely have a Docker networking issue, then. You may have some serious deep-diving with tcpdump and iptables in your future.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.