Metricbeat to elasticsearch (same host with docker) connection failed [forsaken]

[tyteck]

Hi there
I'm pretty new on docker and elasticsearch and I'm trying to install a supervision for my company.
The goal is to have one server hosting both elasticsearch and kibana and many node that are running beats.

I started with elasticsearch, kibana and metricbeat as one mvp installed on my desktop (ubuntu 18.04).
Elasticsearch and kibana are running well, each of them in their containers (I can curl both of them and access to kibana with my browser).
But, metricbeat (container) cannot connect to Elasticsearch.

here is my docker-compose.xml for elasticsearch & kibana

---
version: '3.6'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${TAG}
    hostname: elasticsearch
    container_name: elasticsearch
    ports: 
      - "9200:9200"
      - "9300:9300"
    networks: ['stack']
    volumes:
      - 'es_data:/usr/share/elasticsearch/data'
    healthcheck:
      test: curl -s https://elasticsearch:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

  kibana:
    image: docker.elastic.co/kibana/kibana:${TAG}
    hostname: kibana
    container_name: kibana
    ports: ['5601:5601']
    networks: ['stack']
    depends_on: ['elasticsearch']
    healthcheck:
      test: curl -s https://kibana:5601 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5
  
networks: {stack: {}}
# use docker volume to persist ES data outside of a container.
volumes:
  es_data:

$TAG is set with .env file (TAG=6.4.3).
I run it with a single docker-compose up -d

For metricbeat here is my metricbeat.yml configuration file

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  protocol: "http"

setup.kibana:
  hosts: "kibana:5601"
  protocol: "http"

metricbeat.modules:
- module: system
  metricsets:
    - cpu             # CPU usage
    - load            # CPU load averages
    - memory          # Memory usage
    - network         # Network IO
    - process         # Per process metrics
    - process_summary # Process summary
    - uptime          # System Uptime
    #- core           # Per CPU core usage
    #- diskio         # Disk IO
    #- filesystem     # File system usage for each mountpoint
    #- fsstat         # File system summary metrics
    #- raid           # Raid
    #- socket         # Sockets and connection info (linux only)
  enabled: true
  period: 10s
  processes: ['.*']

  # Configure the metric types that are included by these metricsets.
  cpu.metrics:  ["percentages"]  # The other available options are normalized_percentages and ticks.
  core.metrics: ["percentages"]  # The other available option is ticks.

and the docker-compose.yml

---
version: '3.6'
services:
  metricbeat:
    image: docker.elastic.co/beats/metricbeat:${TAG}
    environment:
      - cluster.name=docker-cluster
    container_name: metricbeat
    #networks: ['stack']
    user: root
    volumes:
      - './metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml'
      - /proc:/hostfs/proc:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /:/hostfs:ro
    healthcheck:
      test: metricbeat test config
      interval: 30s
      timeout: 15s
      retries: 5

Once every "module" is running I got this output with a docker ps -a
79a57ccca6fb docker.elastic.co/beats/metricbeat:6.4.3 "/usr/local/bin/dock…" About a minute ago Up About a minute (healthy)
762ca9b0502e docker.elastic.co/kibana/kibana:6.4.3 "/usr/local/bin/kiba…" About an hour ago Up About an hour (healthy) 0.0.0.0:5601->5601/tcp
c0f38f5212da docker.elastic.co/elasticsearch/elasticsearch:6.4.3 "/usr/local/bin/dock…" About an hour ago Up About an hour (healthy) 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp

Everything seems running
when I attach the container "docker exec -it metricbeat bash" and type "metricbeat export config"
I have this output

metricbeat:
  modules:
  - core:
      metrics:
      - percentages
    cpu:
      metrics:
      - percentages
    enabled: true
    metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    - uptime
    module: system
    period: 10s
    processes:
    - .*
output:
  elasticsearch:
    hosts:
    - elasticsearch:9200
    protocol: http
path:
  config: /usr/share/metricbeat
  data: /usr/share/metricbeat/data
  home: /usr/share/metricbeat
  logs: /usr/share/metricbeat/logs
setup:
  kibana:
    hosts: kibana:5601
    protocol: http

but, still from this container, when I'm trying to connect to the elasticsearch container it fails

metricbeat test output -e -d "*" will give
2018-11-20T11:21:06.506Z INFO instance/beat.go:565 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2018-11-20T11:21:06.506Z DEBUG [beat] instance/beat.go:592 Beat metadata path: /usr/share/metricbeat/data/meta.json
2018-11-20T11:21:06.506Z INFO instance/beat.go:572 Beat UUID: d34f8a92-ebfe-4746-9ef4-63f23d856fa0
2018-11-20T11:21:06.506Z INFO elasticsearch/client.go:163 Elasticsearch url: http://elasticsearch:9200
elasticsearch: http://elasticsearch:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 127.0.0.1
dial up... ERROR dial tcp 127.0.0.1:9200: connect: connection refused

same for

curl http://127.0.0.1:9200/_cat/health
curl: (7) Failed connect to 127.0.0.1:9200; Connection refused

The thing I fail to understand is that when I leave the container curl is accepted

curl http://127.0.0.1:9200/_cat/health
1542713001 11:23:21 docker-cluster yellow 1 1 11 11 0 0 10 0 - 52.4%

Any help will be greatly appreciated

[tyteck]

I gave up with docker and installed it with apt. Everything works fine now