# Unable to access external ElasticSearch Endpoint from within Kibana Docker Container

**URL:** https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714
**Category:** Kibana
**Tags:** docker
**Created:** [November 14, 2018, 5:00pm UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714 "2018-11-14T17:00:12Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![titan1978](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@titan1978](https://discuss.elastic.co/u/titan1978)
#### Post date: [November 14, 2018, 5:00pm UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/1 "2018-11-14T17:00:12Z")

</div>

I'm trying to access an external elasticsearch URL from within my Kibana Docker Container running on an AWS EC2 Instance.

Though I'm able to CURL from the EC2 Instance - the same CURL doesn't work from within the Docker container. Naturally, since my EC2 Instance can connect to the elasticsearch URL, its not a SecurityGroup/Firewall issue.  
Essentially this CURL works in the EC2 Instance but NOT within the Docker Container. Why would that be? Consequently, kibana fails to come up as indicated below with the error of timeout. Interesting is that its not a Connection Refused but a timeout.

> curl [https://elasticsearch.myorg.com](https://elasticsearch.myorg.com)

Dockerfile:

```
FROM docker.elastic.co/kibana/kibana:6.5.0
ENV ELASTICSEARCH_URL=https://elasticsearch.myorg.com
EXPOSE 5601
CMD ["/usr/local/bin/kibana-docker"]

```

This is the error I'm getting from within the docker container logs:

```
{"type":"log","@timestamp":"2018-11-14T16:58:28Z","tags":["status","plugin:logstash@6.4.2","error"],"pid":1,"state":"red","message":"Status changed from red to red - Request Timeout after 30000ms","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2018-11-14T16:59:28Z","tags":["license","warning","xpack"],"pid":1,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 30000ms"}

```

---

<div class="post-metadata">

### Author: ![jarpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jarpy/32/51290_2.png) [@jarpy](https://discuss.elastic.co/u/jarpy)
#### Post date: [November 16, 2018, 4:06am UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/2 "2018-11-16T04:06:11Z")

</div>

What happens if you run:

```auto
docker run --rm byrnedo/alpine-curl https://elasticsearch.myorg.com

```

?

---

<div class="post-metadata">

### Author: ![jarpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jarpy/32/51290_2.png) [@jarpy](https://discuss.elastic.co/u/jarpy)
#### Post date: [November 16, 2018, 4:14am UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/3 "2018-11-16T04:14:52Z")

</div>

And if you explicitly run:

```auto
docker run --rm docker.elastic.co/kibana/kibana:6.5.0 curl https://elasticsearch.myorg.com

```

Does that fail?

What about with your custom image?

---

<div class="post-metadata">

### Author: ![jarpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jarpy/32/51290_2.png) [@jarpy](https://discuss.elastic.co/u/jarpy)
#### Post date: [November 16, 2018, 4:21am UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/4 "2018-11-16T04:21:05Z")

</div>

Actually, the logs from your custom image contain references to `plugin:logstash@6.4.2`, but your Dockerfile is based on Kibana 6.5.0. Did a step fail between rebuilding the custom image and running it?

---

<div class="post-metadata">

### Author: ![titan1978](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@titan1978](https://discuss.elastic.co/u/titan1978)
#### Post date: [November 16, 2018, 4:28am UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/5 "2018-11-16T04:28:52Z")

</div>

So i managed to get this to "work" but using the following command. Found some ancient post in stack overflow where someone wasn't exactly facing this but the options seemed to be a non-binding loopback address call that magically started working:

`docker run --net=host -d -p 5601:5601 my-kibana-image:1`

But the "non-binding loopback" is just jargon to me. If anyone can explain WHY it worked - I would consider the matter truly resolved (the magic behind --net=host)

---

<div class="post-metadata">

### Author: ![jarpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jarpy/32/51290_2.png) [@jarpy](https://discuss.elastic.co/u/jarpy)
#### Post date: [November 19, 2018, 1:46am UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/6 "2018-11-19T01:46:54Z")

</div>

"non-binding loopback" doesn't mean much to me either.

However, `--net=host` connects the container process (Kibana) directly the network interface(s) on the host system, rather than running it in its own network namespace with its own virtual interfaces.

However, we haven't discovered _why_ this is necessary in your environment. You should not have to use `--net=host` to allow Kibana to "see out", like this:

```nohighlight
$ docker run --rm docker.elastic.co/kibana/kibana:6.5.0 curl -s http://example.org | head -n4
<!doctype html>
<html>
<head>
    <title>Example Domain</title>

```

---

<div class="post-metadata">

### Author: ![titan1978](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@titan1978](https://discuss.elastic.co/u/titan1978)
#### Post date: [November 19, 2018, 1:59am UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/7 "2018-11-19T01:59:41Z")

</div>

our EC2 Instances sit behind Bastion Hosts. Could that have something to do with this?  
It would be strange since our "regular" non-docker based kibana run on the same bastion Ec2s without problem

---

<div class="post-metadata">

### Author: ![jarpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jarpy/32/51290_2.png) [@jarpy](https://discuss.elastic.co/u/jarpy)
#### Post date: [November 19, 2018, 2:05am UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/8 "2018-11-19T02:05:19Z")

</div>

It's hard to say. My first guess would be something about the routing rules or netfilter rules on the Docker host. Docker networking works because of a bunch of subtle stuff in netfilter. Just have a look at the output of `iptables -L -v` to see how much trickery is going on! 🙂

Personally, I would start poking around with `tcpdump` to see how far the packets are making it. For example, do they ever leave the Docker host and actually get on the wire?

---

<div class="post-metadata">

### Author: ![titan1978](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@titan1978](https://discuss.elastic.co/u/titan1978)
#### Post date: [November 19, 2018, 2:08am UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/9 "2018-11-19T02:08:22Z")

</div>

Do proxies matter from within Docker Containers?  
Our EC2 based kibana drive off our proxies and work fine. I did try to set the proxies using ENV from within the container but that didnt do anyhting

---

<div class="post-metadata">

### Author: ![jarpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jarpy/32/51290_2.png) [@jarpy](https://discuss.elastic.co/u/jarpy)
#### Post date: [November 19, 2018, 2:16am UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/10 "2018-11-19T02:16:59Z")

</div>

Are you saying that you access Kibana through a proxy, or that Kibana accesses Elasticsearch through a proxy?

Even if it's the latter, I wouldn't expect that to be worked around by `--net=host`. If Kibana needs a proxy to contact Elasticsearch, and it's not correctly configured, it won't matter which network interface it's connected to.

What happens if you run:

```auto
docker run --rm byrnedo/alpine-curl http://example.org

```

?

---

<div class="post-metadata">

### Author: ![titan1978](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@titan1978](https://discuss.elastic.co/u/titan1978)
#### Post date: [November 21, 2018, 5:05pm UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/11 "2018-11-21T17:05:16Z")

</div>

Had to repackage alpine-curl since i could not access the public repo!

Running curl to the elasticsearch endpoint WITH --net=host works perfectly:

> docker run --net=host [artifactory.cloud.myhost.com/alpine-curl](http://artifactory.cloud.myhost.com/alpine-curl) https:/my-elastic-endpoint
> 
> ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b6a50454193f1d90baa77b41d5c2e4f92f7ee13e.png)

Running curl to the elasticsearch endpoint WITHOUT --net=host hangs forever:

> docker run [artifactory.cloud.myhost.com/alpine-curl](http://artifactory.cloud.myhost.com/alpine-curl) https:/my-elastic-endpoint
> 
> ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/a/fae5cf3199fcd2bed4c2c7c2570cecce9b266e00.png)

---

<div class="post-metadata">

### Author: ![jarpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jarpy/32/51290_2.png) [@jarpy](https://discuss.elastic.co/u/jarpy)
#### Post date: [November 22, 2018, 10:33pm UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/12 "2018-11-22T22:33:09Z")

</div>

OK. That's really interesting. I'd say you definitely have a Docker networking issue, then. You may have some serious deep-diving with `tcpdump` and `iptables` in your future.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 20, 2018, 10:33pm UTC](https://discuss.elastic.co/t/unable-to-access-external-elasticsearch-endpoint-from-within-kibana-docker-container/156714/13 "2018-12-20T22:33:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
