Metricbeat - Docker Container stats in Mesos DCOS


(Jim) #1

Does anyone have best practice for sending Elastic Docker stats from containers running in DCOS on CentOS? I'm running Elastic 5.2.2. I see some discussion with using filebeat, but i'm interested in the cpu & mem usage of each container and think I should be using metricbeat instead. Anyone accomplish this and/or have specific guidance to accomplish this? Any info would be much appreciated.


(Carlos Pérez Aradros) #2

Hi @jruzicka,

You may want to have a look to Metricbeat Docker module: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-docker.html, it can retrieve the info you want from Docker.

Also check https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html on how to run metricbeat on top of Docker.

Ideally you would launch a metricbeat container per host, so it can monitor all local containers


(Jim) #3

Thank you. I was able to get metricbeat running on one of our agents in a DCOS cluster. Are you familiar with DCOS and container metrics? For instance I believe I see cgroup system.process.cgroup.memory.mem.usage.bytes on container tasks that contradict docker stats memory values as well as contradicts the mem setting that was allocated to the dcos service.


(Carlos Pérez Aradros) #4

Could you please elaborate? It would be useful if you share the exact metrics & values that seem wrong


(Jim) #5

Sure thing. Thanks for looking into this. I am running DCOS 1.9.2, Elastic 5.2.2

When running "docker stats" on the private agent it returns a number of containers this is one for example:
CONTAINER CPU % MEM USAGE / LIMIT NAME
9bf5e111541f 0.06% 253.8 MiB / 256 MiB mesos-9e706938-*******

I then went into DCOS and reviewed the stdout files for each Docker task running and identified the specific dcos service container that the above mesos-* name value represents. DCOS has 256MiB set for this service. So I was thinking this looks good.

The next step is what I'm not so sure about.
I put a metricbeat on the same private node. The metricbeat.yml is pretty simple it is:

-module: system
metricsets: [process]
cgroups: true
enabled: true
period: 30s
processes: ['.*']

I found this at: https://www.elastic.co/blog/monitoring_container_resource_usage_with_metricbeat

This sends a lot of information into Elastic and I only ran it for a minute.

In Kibana I queried to find the system.process.name = "docker" and excluded other process.name's that might have docker in the value such as: "mesos-docker-ex" and "docker-proxy" and "docker-containe"

The resulting docs I was able to find the service i'm interested in in the field: system.process.cmdline. On this same doc the field: system.process.cgroup.memory.mem.usage.bytes has the value: 538,157,056

I was expecting to see 253.8MB as shown in docker stats. Maybe i'm not understanding the cgroup values correctly.

I found this post which makes me think there might be an issue with DCOS memory limits on Docker containers.
https://jira.mesosphere.com/browse/DCOS_OSS-1501

I also read that Docker will take more memory then allocated if you do not specify a memory max in your docker-compose.yml - which I have not.

Any information would be greatly appreciated.


(Carlos Pérez Aradros) #6

I think it's better if you use docker memory metricset, https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-docker.html like:

- module: docker
  metricsets:
    - memory

Check this for a list of fields you should get: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-docker-memory.html


(Jim) #7

Thank you, I have this working as you recommended. I feel like with metricbeat I can get just as much detail as I can with DockBeat. Is there some benefit of Dockbeat that maybe i'm missing? https://github.com/Ingensi/dockbeat


(Carlos Pérez Aradros) #8

I haven't used Dockbeat but from what I see in https://github.com/Ingensi/dockbeat/blob/develop/docs/fields.asciidoc I think most of it is already covered by Metricbeat


(system) #9

This topic was automatically closed after 21 days. New replies are no longer allowed.