I have set up monitoring Docker containers with Metricbeat and after a couple of days I have spotted suspicious regular peaks in connections to Docker's unix socket. Netstat pointed to Metricbeat. I have enabled debug mode in Docker and noticed that all the requests were to container stats API which is kinda expected but instead of expected one connection per container there were actually four. I have gone through the Metricbeat's code and I have noticed that each of the four metrics: cpu, memory, diskio and network fetch the container's stats independently although container stats API returns all the metrics in one response. So, if I got everything right, current implementation, in default setup establishes four times more connections to Docker daemon than necessary, and if next metrics are added by Docker the number will go up.
I am wandering then if there is already a plan to optimize this inefficiency in some future version of Metricbeat?
Hi @localghost,
I would say your assesment is correct. Each metricset acts independently and gathers its own metrics, in most cases this requires different calls, it's not the case with Docker.
In general, although it's not the optimal, it should not cause any issue. Is it impacting your system somehow?
Best regards
Hi @exekias,
It is not a problem when looked at individually but we are trying to minimise resource consumption on our testing environment, and having burst of around 40 to 50 connections to docker daemon every 10 seconds, reaching for stats which is marked as inefficient even in beats code:
This is currently very inefficient as docker calculates the average for each request,
is yet another issue that we must count in when on one hand we want to limit the resources for docker daemon and on the other hand we want to keep it responsive.
I am not saying this is a bug and I am fully aware that it will add complexity to the beats code, but I hope this could be improved in the future version of beats so that if e.g. docker starts exposing more metrics they could be retrieved by metricbeat in one shot.
Agreed,
In any case, you could open an enhancement request for this here: https://github.com/elastic/beats/issues
That way we can easily keep track of the issue and try to improve current status.
Best regards
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.