I'd like to use metricbeat to collect cpu and memory usage so that I can better monitor running applications and be alerted when an application is crashing in a loop. I used this code to start metricbeat in docker in the lxc container but it doesnt seem to report cpu usage and memory usage correctly.
thanks for the quick answer. Yes that is my use case. You have a lot less overhead than running a complete Ubuntu VM. In Proxmox you have a summary for every LXC with CPU usage and used/free/total memory. Sometimes an application crashes when auto updating with containerr watchtowerr and uses a lot of CPU. I’d like to get notified when this happens. In addition I’d like to create dashboards for all LXC containers.
To test I started stress testing an LXC (CPU usage should be 1) but with the host cpu usage field I could only get the underlying host cpu usage of 26% (1 of 4 cores is mapped to the LXC)
I do the same on my homelab, Proxmox and a bunch of LXC, the only VM running is Home Assistant.
Could you share your metricbeat.yml (remember to redact any sensitive information)?
Just to double check I understood what you want to do: you want to run Metricbeat inside a docker container and Metricbeat should get the metrics from the host (the LXC container). Is that it?
How did you run your stress test? Did you run it directly on the LXC container or as a docker container?
Your Metricbeat is configured to collect metrics from docker containers, so if your containers are idle and you're stressing the host, you won't see this in the metrics.
I went the lazy way to try this out and deployed the Elastic-Agent (it uses Metricbeat to collect the data) to my LXC container configured to get host metrics and what it got is pretty similar to what I see in Proxmox. Based on my charts it seems Proxmox is showing the "user" bit from what Metricbeat collected.
To collect the host metrics, you need to enable the system module.
In case you're curious, here are the metrics are reported by Proxmox and Metricbeat:
That looks just the way I want it to have @TiagoQueiroz . That explains why I didn’t get LXC usage when I’m only listening for docker. Could you please post your configuration and steps of installing the apm agent on the LXC container? Do I have to configure anything on the elastic server?
I meant the Elastic-Agent, not the APM Agents. On Kibana you go to Management -> Fleet -> Add agent. Then the UI guides you through the process.
Simply put (that's a huge over-simplification), the Elastic-Agent is a supervisor that deploys Beats for you. Everything is managed through Kibana, so no need to deal with YAMLs and low level details you mostly don't need to worry about.
Here is my config. Bear in mid it's auto-generated, so the keys are in no specific order, and there are loads of extra stuff/transformations. In the end of the day, all entries are the system module with specific configuration for each metricset.
Do you know what I'm doing wrong? The fleet server says:
18:08:38.520 elastic_agent [elastic_agent][info] No events received within 10m0s, restarting watch call
18:18:48.520 elastic_agent [elastic_agent][info] No events received within 10m0s, restarting watch call
That is odd. It looks like the Beats cannot communicate with Elasticsearch.
Go to the instance where the Elastic-Agent is installed. If you installed on a Linux host, it will be on /opt/Elastic/Agent, then run elastic-agent diagnostics collect . This will generate a zip file with all the configs and logs from Elastic-Agent and Beats.
Look for error message in the logs, and check the URLs and credentials configured for Beats are correct.
I don't know this Docker compose you're using, there is a chance some of its default configurations use a internal docker hostname or some ports are not fully exposed.
But if you want to host yourself, you can just download the tar.gz version of Elasticsearch and Kibana and set them up "manually". There are a number of steps to follow, but it's not hard, keep the default security/SSL settings and follow the documentation.
By having everything running directly on your machine it should make debugging connection issues easy.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.