Metricbeat unable to connect to elasticsearch

Invictus · April 13, 2018, 8:59am

Hi,

i have a problem with metricbeat collecting container-data and send them to elasticsearch.

Why does metricbeat try to connect to elasticsearch via http://elasticsearch:9200 even though I set the output to the container elasticsearch-coordinating:9200?
Can't the container connect to the unix:///var/run/docker.sock?

Thanks for some advice

sudo docker ps -a

b07f73f2159f        dockerelk_elasticsearch                "/usr/local/bin/dock…"   About an hour ago   Up 17 minutes       9200/tcp, 0.0.0.0:32890->9300/tcp                dockerelk_elasticsearch_2
74f6fef3e23a        dockerelk_elasticsearch                "/usr/local/bin/dock…"   About an hour ago   Up 17 minutes       9200/tcp, 0.0.0.0:32891->9300/tcp                dockerelk_elasticsearch_3
76e7ec2ff476        dockerelk_elasticsearch-master         "/usr/local/bin/dock…"   About an hour ago   Up 17 minutes       9200/tcp, 0.0.0.0:32888->9300/tcp                dockerelk_elasticsearch-master_2
5751436e7a98        dockerelk_metricbeat                   "/usr/local/bin/dock…"   19 hours ago        Up 17 minutes                                                        dockerelk_metricbeat_1
26858c6a7221        dockerelk_elasticsearch-master         "/usr/local/bin/dock…"   20 hours ago        Up 17 minutes       9200/tcp, 0.0.0.0:32889->9300/tcp                dockerelk_elasticsearch-master_1
33e1661cad34        dockerelk_elasticsearch-coordinating   "/usr/local/bin/dock…"   20 hours ago        Up 17 minutes       0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elasticsearch-coordinating
4dcc264e97d9        dockerelk_logstash                     "/usr/local/bin/dock…"   23 hours ago        Up 17 minutes       5044/tcp, 0.0.0.0:5000->5000/tcp, 9600/tcp       dockerelk_logstash_1
d2cd4cbfd4a4        dockerelk_kibana                       "/bin/bash /usr/loca…"   23 hours ago        Up 17 minutes       0.0.0.0:5601->5601/tcp                           dockerelk_kibana_1
a899f42c5a45        dockerelk_elasticsearch                "/usr/local/bin/dock…"   23 hours ago        Up 17 minutes       9200/tcp, 0.0.0.0:32892->9300/tcp                dockerelk_elasticsearch_1

The logs that Metricbeat is writing are the following:

2018-04-13T07:31:07.639Z        INFO    [monitoring]    log/log.go:133  Uptime: 4m33.349175494s
2018-04-13T07:31:07.639Z        INFO    [monitoring]    log/log.go:110  Stopping metrics logging.
2018-04-13T07:31:08.942Z        INFO    instance/beat.go:468    Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2018-04-13T07:31:08.942Z        INFO    instance/beat.go:475    Beat UUID: 14e90688-2d28-449a-a972-509d290b1257
2018-04-13T07:31:08.942Z        INFO    instance/beat.go:213    Setup Beat: metricbeat; Version: 6.2.3
2018-04-13T07:31:11.943Z        INFO    add_cloud_metadata/add_cloud_metadata.go:297    add_cloud_metadata: hosting provider type not detected.
2018-04-13T07:31:11.943Z        INFO    elasticsearch/client.go:145     Elasticsearch url: http://elasticsearch:9200
2018-04-13T07:31:11.943Z        INFO    pipeline/module.go:76   Beat name: 5751436e7a98
2018-04-13T07:31:11.944Z        INFO    instance/beat.go:301    metricbeat start running.
2018-04-13T07:31:11.944Z        INFO    [monitoring]    log/log.go:97   Starting metrics logging every 30s
2018-04-13T07:31:11.945Z        INFO    cfgfile/reload.go:127   Config reloader started
2018-04-13T07:31:11.948Z        INFO    cfgfile/reload.go:258   Starting 4 runners ...
2018-04-13T07:31:11.948Z        INFO    cfgfile/reload.go:219   Loading of config files completed.
2018-04-13T07:31:13.971Z        ERROR   pipeline/output.go:74   Failed to connect: Get http://elasticsearch:9200: dial tcp 172.18.0.7:9200: getsockopt: connection refused
2018-04-13T07:31:15.973Z        ERROR   pipeline/output.go:74   Failed to connect: Get http://elasticsearch:9200: dial tcp 172.18.0.6:9200: getsockopt: connection refused
2018-04-13T07:31:19.974Z        ERROR   pipeline/output.go:74   Failed to connect: Get http://elasticsearch:9200: dial tcp 172.18.0.7:9200: getsockopt: connection refused
2018-04-13T07:31:27.976Z        ERROR   pipeline/output.go:74   Failed to connect: Get http://elasticsearch:9200: dial tcp 172.18.0.7:9200: getsockopt: connection refused
2018-04-13T07:31:41.946Z        INFO    [monitoring]    log/log.go:124  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":60},"total":{"ticks":140,"time":148,"value":140},"user":{"ticks":80,"time":88}},"info":{"ephemeral_id":"50326811-cc75-4f97-950e-268ac58fd19e","uptime":{"ms":33062}},"memstats":{"gc_next":7427872,"memory_alloc":6009496,"memory_total":15975344,"rss":35540992}},"libbeat":{"config":{"module":{"running":4,"starts":4},"reloads":1},"output":{"type":"elasticsearch"},"pipeline":{"clients":8,"events":{"active":164,"filtered":34,"published":164,"retry":66,"total":198}}},"metricbeat":{"docker":{"container":{"events":27,"success":27},"cpu":{"events":27,"success":27},"diskio":{"events":27,"success":27},"info":{"events":3,"success":3},"memory":{"events":27,"success":27},"network":{"events":27,"success":27}},"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":37,"success":37},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":6,"success":6},"process":{"events":3,"success":3},"process_summary":{"events":3,"success":3},"uptime":{"events":1,"success":1}}},"system":{"cpu":{"cores":2},"load":{"1":9.35,"15":1.81,"5":3.83,"norm":{"1":4.675,"15":0.905,"5":1.915}}}}}}

docker-compose.yml

  metricbeat:
    build:
      context: metricbeat/
    user: root
    volumes:
      - ./metricbeat/config/metricbeat.yml:/usr/share/metricbeat/config/metricbeat.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - elk
    links:
      - elasticsearch-coordinating
    depends_on:
      - elasticsearch-master
      - elasticsearch-coordinating

metricbeat.yml

metricbeat.modules:
- module: docker
  metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
    - memory
    - network
  hosts: ["unix:///var/run/docker.sock"]
  enable: true
  period: 10s

output.elasticsearch:
  hosts: ["elasticsearch-coordinating:9200"]

andrewkroh · April 13, 2018, 3:48pm

The path to the config file looks wrong. When using our official container images the path is /usr/share/metricbeat/metricbeat.yml.

References:

Running Metricbeat on Docker | Metricbeat Reference [6.2] | Elastic

Invictus · April 16, 2018, 11:36am

Hi Andrew,

thanks for your reply. Actually every container has the setting usr/share/CONTAINER/config/*.yml.
I removed config but this doesnt fix the problem.

I suggest it has something to do with the network bridge, maybe that metricbeat doesnt connect to network: elk but I dont now how to check this?

andrewkroh · April 16, 2018, 4:03pm

Are you using the official Metricbeat container image? Did you customize anything for the CMD or ENTRYPOINT? I don't see how metricbeat could be using your configuration file from /usr/share/metricbeat/config/metricbeat.yml unless you did. By default it's going to be reading /usr/share/metricbeat/metricbeat.yml unless you set -c or -path.config on the CLI.

That wouldn't explain why it's connecting to elasticsearch instead of elasticsearch-coordinating.

But to debug this more I would docker exec into the container and try a looking at ./metricbeat export config and ./metricbeat test output -e -d "*".

In addition to this you could add logging.level: debug to config and see if there's any more clues in the log output.

Invictus · April 19, 2018, 6:55am

Ok thanks for your advice.
I checked the logs that have been written. Now all errors are gone and it seems that he is connecting to the elasticsearch-cooridinating container. But there are no metrics transmitted.

2018-04-19T06:33:57.241Z        INFO    instance/beat.go:468    Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2018-04-19T06:33:57.241Z        INFO    instance/beat.go:475    Beat UUID: 3fcc385f-dcd2-4566-849e-1995343dfbe7
2018-04-19T06:33:57.241Z        INFO    instance/beat.go:213    Setup Beat: metricbeat; Version: 6.2.3
2018-04-19T06:33:57.241Z        INFO    elasticsearch/client.go:145     Elasticsearch url: http://elasticsearch-coordinating:9200
2018-04-19T06:33:57.242Z        INFO    pipeline/module.go:76   Beat name: metricbeat
2018-04-19T06:33:57.242Z        INFO    instance/beat.go:301    metricbeat start running.
2018-04-19T06:33:57.242Z        INFO    [monitoring]    log/log.go:97   Starting metrics logging every 30s
2018-04-19T06:33:57.242Z        INFO    cfgfile/reload.go:127   Config reloader started
2018-04-19T06:33:57.242Z        INFO    cfgfile/reload.go:219   Loading of config files completed.
2018-04-19T06:34:27.244Z        INFO    [monitoring]    log/log.go:124  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":24},"total":{"ticks":60,"time":64,"value":60},"user":{"ticks":40,"time":40}},"info":{"ephemeral_id":"0c60b26d-2f95-420e-9cde-16091860d808","uptime":{"ms":30031}},"memstats":{"gc_next":4194304,"memory_alloc":2197840,"memory_total":3821904,"rss":29904896}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":2},"load":{"1":0.61,"15":2.17,"5":0.93,"norm":{"1":0.305,"15":1.085,"5":0.465}}}}}}

./metricbeat export config

metricbeat:
  config:
    modules:
      enabled: true
      path: home/user/docker-elk/metricbeat/modules.d/*.yml
      period: 10s
output:
  elasticsearch:
    hosts:
    - elasticsearch-coordinating:9200
path:
  config: /usr/share/metricbeat
  data: /usr/share/metricbeat/data
  home: /usr/share/metricbeat
  logs: /usr/share/metricbeat/logs

./metricbeat test output -e -d "*"

2018-04-19T06:46:52.705Z        INFO    instance/beat.go:468    Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2018-04-19T06:46:52.706Z        DEBUG   [beat]  instance/beat.go:495    Beat metadata path: /usr/share/metricbeat/data/meta.json
2018-04-19T06:46:52.706Z        INFO    instance/beat.go:475    Beat UUID: 3fcc385f-dcd2-4566-849e-1995343dfbe7
2018-04-19T06:46:52.706Z        INFO    elasticsearch/client.go:145     Elasticsearch url: http://elasticsearch-coordinating:9200
elasticsearch: http://elasticsearch-coordinating:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.18.0.7
    dial up... OK
  TLS... WARN secure connection disabled
2018-04-19T06:46:52.708Z        DEBUG   [elasticsearch] elasticsearch/client.go:666     ES Ping(url=http://elasticsearch-coordinating:9200)
2018-04-19T06:46:52.711Z        DEBUG   [elasticsearch] elasticsearch/client.go:689     Ping status code: 200
2018-04-19T06:46:52.711Z        INFO    elasticsearch/client.go:690     Connected to Elasticsearch version 6.2.2
  talk to server... OK
  version: 6.2.2

But when I set the command metricbeat modules list there are no modules listed. I think that is a problem, where can that be fixed?

[root@metricbeat metricbeat]# metricbeat modules list
Enabled:

Disabled:

Best regards and thanks in advance

system · May 17, 2018, 6:55am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.