Hi,
i have a problem with metricbeat collecting container-data and send them to elasticsearch.
Why does metricbeat try to connect to elasticsearch via http://elasticsearch:9200 even though I set the output to the container elasticsearch-coordinating:9200?
Can't the container connect to the unix:///var/run/docker.sock?
Thanks for some advice 
sudo docker ps -a
b07f73f2159f dockerelk_elasticsearch "/usr/local/bin/dockā¦" About an hour ago Up 17 minutes 9200/tcp, 0.0.0.0:32890->9300/tcp dockerelk_elasticsearch_2
74f6fef3e23a dockerelk_elasticsearch "/usr/local/bin/dockā¦" About an hour ago Up 17 minutes 9200/tcp, 0.0.0.0:32891->9300/tcp dockerelk_elasticsearch_3
76e7ec2ff476 dockerelk_elasticsearch-master "/usr/local/bin/dockā¦" About an hour ago Up 17 minutes 9200/tcp, 0.0.0.0:32888->9300/tcp dockerelk_elasticsearch-master_2
5751436e7a98 dockerelk_metricbeat "/usr/local/bin/dockā¦" 19 hours ago Up 17 minutes dockerelk_metricbeat_1
26858c6a7221 dockerelk_elasticsearch-master "/usr/local/bin/dockā¦" 20 hours ago Up 17 minutes 9200/tcp, 0.0.0.0:32889->9300/tcp dockerelk_elasticsearch-master_1
33e1661cad34 dockerelk_elasticsearch-coordinating "/usr/local/bin/dockā¦" 20 hours ago Up 17 minutes 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp elasticsearch-coordinating
4dcc264e97d9 dockerelk_logstash "/usr/local/bin/dockā¦" 23 hours ago Up 17 minutes 5044/tcp, 0.0.0.0:5000->5000/tcp, 9600/tcp dockerelk_logstash_1
d2cd4cbfd4a4 dockerelk_kibana "/bin/bash /usr/locaā¦" 23 hours ago Up 17 minutes 0.0.0.0:5601->5601/tcp dockerelk_kibana_1
a899f42c5a45 dockerelk_elasticsearch "/usr/local/bin/dockā¦" 23 hours ago Up 17 minutes 9200/tcp, 0.0.0.0:32892->9300/tcp dockerelk_elasticsearch_1
The logs that Metricbeat is writing are the following:
2018-04-13T07:31:07.639Z INFO [monitoring] log/log.go:133 Uptime: 4m33.349175494s
2018-04-13T07:31:07.639Z INFO [monitoring] log/log.go:110 Stopping metrics logging.
2018-04-13T07:31:08.942Z INFO instance/beat.go:468 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2018-04-13T07:31:08.942Z INFO instance/beat.go:475 Beat UUID: 14e90688-2d28-449a-a972-509d290b1257
2018-04-13T07:31:08.942Z INFO instance/beat.go:213 Setup Beat: metricbeat; Version: 6.2.3
2018-04-13T07:31:11.943Z INFO add_cloud_metadata/add_cloud_metadata.go:297 add_cloud_metadata: hosting provider type not detected.
2018-04-13T07:31:11.943Z INFO elasticsearch/client.go:145 Elasticsearch url: http://elasticsearch:9200
2018-04-13T07:31:11.943Z INFO pipeline/module.go:76 Beat name: 5751436e7a98
2018-04-13T07:31:11.944Z INFO instance/beat.go:301 metricbeat start running.
2018-04-13T07:31:11.944Z INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-04-13T07:31:11.945Z INFO cfgfile/reload.go:127 Config reloader started
2018-04-13T07:31:11.948Z INFO cfgfile/reload.go:258 Starting 4 runners ...
2018-04-13T07:31:11.948Z INFO cfgfile/reload.go:219 Loading of config files completed.
2018-04-13T07:31:13.971Z ERROR pipeline/output.go:74 Failed to connect: Get http://elasticsearch:9200: dial tcp 172.18.0.7:9200: getsockopt: connection refused
2018-04-13T07:31:15.973Z ERROR pipeline/output.go:74 Failed to connect: Get http://elasticsearch:9200: dial tcp 172.18.0.6:9200: getsockopt: connection refused
2018-04-13T07:31:19.974Z ERROR pipeline/output.go:74 Failed to connect: Get http://elasticsearch:9200: dial tcp 172.18.0.7:9200: getsockopt: connection refused
2018-04-13T07:31:27.976Z ERROR pipeline/output.go:74 Failed to connect: Get http://elasticsearch:9200: dial tcp 172.18.0.7:9200: getsockopt: connection refused
2018-04-13T07:31:41.946Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":60},"total":{"ticks":140,"time":148,"value":140},"user":{"ticks":80,"time":88}},"info":{"ephemeral_id":"50326811-cc75-4f97-950e-268ac58fd19e","uptime":{"ms":33062}},"memstats":{"gc_next":7427872,"memory_alloc":6009496,"memory_total":15975344,"rss":35540992}},"libbeat":{"config":{"module":{"running":4,"starts":4},"reloads":1},"output":{"type":"elasticsearch"},"pipeline":{"clients":8,"events":{"active":164,"filtered":34,"published":164,"retry":66,"total":198}}},"metricbeat":{"docker":{"container":{"events":27,"success":27},"cpu":{"events":27,"success":27},"diskio":{"events":27,"success":27},"info":{"events":3,"success":3},"memory":{"events":27,"success":27},"network":{"events":27,"success":27}},"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":37,"success":37},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":6,"success":6},"process":{"events":3,"success":3},"process_summary":{"events":3,"success":3},"uptime":{"events":1,"success":1}}},"system":{"cpu":{"cores":2},"load":{"1":9.35,"15":1.81,"5":3.83,"norm":{"1":4.675,"15":0.905,"5":1.915}}}}}}
docker-compose.yml
metricbeat:
build:
context: metricbeat/
user: root
volumes:
- ./metricbeat/config/metricbeat.yml:/usr/share/metricbeat/config/metricbeat.yml:ro
- /var/run/docker.sock:/var/run/docker.sock
networks:
- elk
links:
- elasticsearch-coordinating
depends_on:
- elasticsearch-master
- elasticsearch-coordinating
metricbeat.yml
metricbeat.modules:
- module: docker
metricsets:
- container
- cpu
- diskio
- healthcheck
- info
- memory
- network
hosts: ["unix:///var/run/docker.sock"]
enable: true
period: 10s
output.elasticsearch:
hosts: ["elasticsearch-coordinating:9200"]