How to use Metricbeat to monitor Docker Swarm containers, hosts and external services

[hamid.haghshenas]

We need to monitor:

• a number of Docker Swarm services, each may have multiple containers on multiple hosts (A host may run several containers of the same service),
• hosts running Docker Swarm, and
• Some external services like MySQL which are not on Docker Swarm

First question:

I've read in other topics to run an instance of Metricbeat on each host. But, can I monitor all the above with one metricbeat service with mode: global (i.e., one container per host)?

Assuming the answer is "no" and I cannot monitor MySQL with such a setting, I decided to have two Swarm services: metricbeat_node and metricbeat_single. The former runs with mode: global and has system and docker modules. The latter is a singleton service and has mysql module. If I'm wrong, what should I do for this?

Second question:

If the above settings is accepted, then which of the two Metricbeat services should monitor other Swarm services? Say, I have a service my_app running uWSGI and may have many replicas. Is it better to monitor it using metricbeat_node (i.e., monitor each host's containers with a Metricbeat container running on that host), or metricbeat_single (i.e., monitor all of them centrally)? Which is better and why?

[ruflin]

It's recommend to run your Metricbeat container as a side car and mount the correct volumes: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html If you do that, you can monitor all services form 1 container + the host + the containers.

[hamid.haghshenas]

Thanks @ruflin; I read your answer multiple times and read the link you provided completely, but didn't get the point.

By 'side car' do you mean that I need separate Metricbeat containers for monitoring different thins? That is, I need one Metricbeat container on each node to monitor the hosts, another container to monitor my external MySQL, another set of containers to monitor NGINX, Redis, etc.?

[ruflin]

You need only 1 container per docker host to monitor the host, all containers and the services inside. So if you have host machines, you will need 3 metricbeat instances.