Docker Filebeat can't connect to Docker bridge network despite ELK being able to

Elastic Stack Beats
1

My setup

I'm using a slight modification* of the master branch of docker-elk via Docker for Ubuntu 16.04 v18.06.1-ce. The docker-elk repo is currently based on the Elastic stack v6.3.2, and specifically uses the following images:

For convenience: link to the docker.elastic.co repository.

Modification to master branch of ELK stack

* The modification is in the networking. In the docker-compose.yml, instead of setting networks.elk.driver: bridge to delegate creation of the network to the docker-compose script, I use networks.elk.external: true and manually create the bridge network myself:

sudo docker network create --driver=bridge elk

I do it like this to ensure that the network does not become namespaced based on the project folder (namespacing is not consistent between macOS and Linux, as the former strips punctuation from the folder name, while the latter doesn't).

Issue

All the containers in the Docker ELK stack can individually use this network without complaint. However, when I run a separate Filebeat Docker container, I get an error despite using what I believe to be sensible configuration:

#  docker-compose.yml
version: '3'

services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:6.3.2
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro,delegated
    networks:
      - elk
networks:
  elk:
    external: true

# filebeat.yml
output:
  elasticsearch:
    hosts:
      - elk:9200
setup.kibana:
  host: elk:5601
# 'input' removed for brevity

Error log:

|filebeat_1  | 2018-08-28T16:20:29.684Z|ERROR|pipeline/output.go:74|Failed to connect: Get http://elk:9200: lookup elk on 127.0.0.11:53: no such host|
|---|---|---|---|
|filebeat_1  | 2018-08-28T16:20:29.690Z|WARN|transport/tcp.go:36|DNS lookup failure "elk": lookup elk on 127.0.0.11:53: no such host|

Debug

sudo docker network ls confirms that a network named elk exists:

NETWORK ID          NAME                DRIVER              SCOPE
05c3be42ce0e        bridge              bridge              local
34d783fd620f        elk                 bridge              local
ca6aa70af0af        host                host                local
158a5287a79e        none                null                local

sudo docker network inspect elk seems to suggest that Filebeat is on-board:

[
    {
        "Name": "elk",
        "Id": "34d783fd620fdde9034738106e5e1a42bbc2d281e70d8352e941ea26f3c1d7eb",
        "Created": "2018-08-28T15:41:36.62988126+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.23.0.0/16",
                    "Gateway": "172.23.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "4c99023d55a6d75bb9b145ba433391604d207732d44b664ec2622186916ec48f": {
                "Name": "docker-elk_elasticsearch_1",
                "EndpointID": "a93b38110b6a02138fe9b3af8bc60fe9ee0c82d8a279fbb3244c78b2f5a22d5e",
                "MacAddress": "02:42:ac:17:00:02",
                "IPv4Address": "172.23.0.2/16",
                "IPv6Address": ""
            },
            "73c940b97e13fecc2eabdd9ef38294e4e4b4cb9b8c0833cdeec34eb0c3ddffac": {
                "Name": "docker-elk_kibana_1",
                "EndpointID": "c5054bd15fea6bae3b64215fc84c89273aa4c3e9651ee443789ce4f8c3e4e034",
                "MacAddress": "02:42:ac:17:00:03",
                "IPv4Address": "172.23.0.3/16",
                "IPv6Address": ""
            },
            "ac5f2ccb9d98b23e5035451e638200a261ad6357c7d14aeab3129b7ff7fe19bc": {
                "Name": "docker-filebeat_filebeat_1",
                "EndpointID": "ddeb3cb0e1e76fdd4bb73ce4a9cc3b7bd05997473e27f373dbd79b451c444e75",
                "MacAddress": "02:42:ac:17:00:04",
                "IPv4Address": "172.23.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

... Yet, when opening a terminal into the Docker Filebeat container, I don't find any evidence of connectivity:

bash-4.2$ curl http://localhost:9200
curl: (7) Failed to connect to ::1: Cannot assign requested address
bash-4.2$ curl http://localhost:5601
curl: (7) Failed to connect to ::1: Cannot assign requested address
bash-4.2$ curl http://elk:5601
curl: (6) Could not resolve host: elk; Unknown error
bash-4.2$ curl http://elk:9200
curl: (6) Could not resolve host: elk; Unknown error
bash-4.2$ curl http://localhost:5601
curl: (7) Failed to connect to ::1: Cannot assign requested address

Any inspiration?

2

Are you sure about the elasticsearch endpoint? Have your tried run the container or image on the same network and ping the different hosts.

Each container has it's own IPv4 address and therefore should have it's own hostname within the virtual network (docker itself provides the DNS service for the containers).

E.g. you know the ip addresses. Run bash in the filebeat container and run nslookup 172.23.0.2. This should print the hostname of elasticsearch.
Container names depend on the home directory of the docker-compose file, plus some ID. I guess you have to configure filebeat to connect to http://elasticsearch:9200 and http://kibana:5601.

1 Like
3

Are you sure about the elasticsearch endpoint?

I guess not. I think I may have inherited the elk:<port> idea from somewhere, but looking back, it appears that on macOS, I was using a feature that doesn't exist on Linux (as far as I understand), host.docker.internal:

output.elasticsearch.hosts: ['host.docker.internal:9200']
setup.kibana.host: "host.docker.internal:5601"

Have your tried run the container or image on the same network and ping the different hosts.

They are all on the same elk network, but my issue I suppose is not knowing the endpoint to ping.

... However, your suggestion worked!

I shall note that nslookup wasn't installed in my container, so I wasn't able to get visibility of hostnames, but that was no problem. Reconfiguring filebeat.yml as you recommended did the trick:

output.elasticsearch.hosts: ['http://elasticsearch:9200']
setup.kibana.host: http://kibana:5601

The startup logs looked like this ("Elasticsearch url" seems to simply echo whatever was configured in output.elasticsearch.hosts, as last time it said "http://elk:9200"):

filebeat_1  | 2018-08-29T08:40:15.640Z	INFO	elasticsearch/client.go:145	Elasticsearch url: http://elasticsearch:9200
...
filebeat_1  | 2018-08-29T08:40:18.901Z	INFO	elasticsearch/client.go:690	Connected to Elasticsearch version 6.3.2

Once running bash inside the filebeat container, trying either of the following curls gave a HTTP status 200 and returned a body:

curl http://elasticsearch:9200
curl http://kibana:5601

Thank you very much! Quality support again from the Elastic Team.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.