Does anyone have best practice for sending Elastic Docker stats from containers running in DCOS on CentOS? I'm running Elastic 5.2.2. I see some discussion with using filebeat, but i'm interested in the cpu & mem usage of each container and think I should be using metricbeat instead. Anyone accomplish this and/or have specific guidance to accomplish this? Any info would be much appreciated.
Hi @jruzicka,
You may want to have a look to Metricbeat Docker module: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-docker.html, it can retrieve the info you want from Docker.
Also check https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html on how to run metricbeat on top of Docker.
Ideally you would launch a metricbeat container per host, so it can monitor all local containers
Thank you. I was able to get metricbeat running on one of our agents in a DCOS cluster. Are you familiar with DCOS and container metrics? For instance I believe I see cgroup system.process.cgroup.memory.mem.usage.bytes on container tasks that contradict docker stats memory values as well as contradicts the mem setting that was allocated to the dcos service.
Could you please elaborate? It would be useful if you share the exact metrics & values that seem wrong
Sure thing. Thanks for looking into this. I am running DCOS 1.9.2, Elastic 5.2.2
When running "docker stats" on the private agent it returns a number of containers this is one for example:
CONTAINER CPU % MEM USAGE / LIMIT NAME
9bf5e111541f 0.06% 253.8 MiB / 256 MiB mesos-9e706938-*******
I then went into DCOS and reviewed the stdout files for each Docker task running and identified the specific dcos service container that the above mesos-* name value represents. DCOS has 256MiB set for this service. So I was thinking this looks good.
The next step is what I'm not so sure about.
I put a metricbeat on the same private node. The metricbeat.yml is pretty simple it is:
-module: system
metricsets: [process]
cgroups: true
enabled: true
period: 30s
processes: ['.*']
I found this at: https://www.elastic.co/blog/monitoring_container_resource_usage_with_metricbeat
This sends a lot of information into Elastic and I only ran it for a minute.
In Kibana I queried to find the system.process.name = "docker" and excluded other process.name's that might have docker in the value such as: "mesos-docker-ex" and "docker-proxy" and "docker-containe"
The resulting docs I was able to find the service i'm interested in in the field: system.process.cmdline. On this same doc the field: system.process.cgroup.memory.mem.usage.bytes has the value: 538,157,056
I was expecting to see 253.8MB as shown in docker stats. Maybe i'm not understanding the cgroup values correctly.
I found this post which makes me think there might be an issue with DCOS memory limits on Docker containers.
https://jira.mesosphere.com/browse/DCOS_OSS-1501
I also read that Docker will take more memory then allocated if you do not specify a memory max in your docker-compose.yml - which I have not.
Any information would be greatly appreciated.
I think it's better if you use docker memory metricset, https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-docker.html like:
- module: docker
metricsets:
- memory
Check this for a list of fields you should get: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-docker-memory.html
Thank you, I have this working as you recommended. I feel like with metricbeat I can get just as much detail as I can with DockBeat. Is there some benefit of Dockbeat that maybe i'm missing? https://github.com/Ingensi/dockbeat
I haven't used Dockbeat but from what I see in https://github.com/Ingensi/dockbeat/blob/develop/docs/fields.asciidoc I think most of it is already covered by Metricbeat
This topic was automatically closed after 21 days. New replies are no longer allowed.