Hi,
in the Docs is a run command of docker with which you should be able to Monitor the host machine (with the System Module), but It seems to not be working. In a reddit thread nobody seems to be able to give a working example, despite the one in the Docs.
Please tell me if this is a Bug or give me an working Configuration.
Have a nice Day 
.
Can you provide more context?
What is your host machine? Do you have anything in metricbeat logs?
First Thanks for looking at my Problem.
I want to get the Procesess of the host not of my container.
My host machine is an ubuntu 22.04.
Docker version 20.10.23, build 7155243.
My Metricbeat.yml:
metricbeat.config:
modules:
reload.enabled: false
# Modules configuration
metricbeat.modules:
# System Module
- module: system
metricsets:
- process
enabled: true
period: 30s
processes: ['.*']
# Configure the mount point of the host’s filesystem for use in monitoring a host from within a container
hostfs: "/hostfs"
# Configure the metric types that are included by these metricsets.
cpu.metrics: ["percentages","normalized_percentages"]
core.metrics: ["percentages"]
# Enable collection of cgroup metrics from processes on Linux.
process.cgroups.enabled: true
# Outputs
output.elasticsearch:
enabled: true
# Array of hosts to connect to.
# Scheme and port can be left out and will be set to the default (http and 9200)
# In case you specify and additional path, the scheme is required: http://localhost:9200/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
hosts: ["172.17.0.1:9200"]#the host machine with running elasticsearch
# Protocol - either `http` (default) or `https`.
protocol: "http"
username: "elastic"
password: "myPassword"
docker run:
sudo docker run \
--mount type=bind,source=/proc,target=/hostfs/proc,readonly \
--mount type=bind,source=/sys/fs/cgroup,target=/hostfs/sys/fs/cgroup,readonly \
--mount type=bind,source=/,target=/hostfs,readonly \
--mount type=bind,source=/metricbeat.yml,target=/usr/share/metricbeat/metricbeat.yml,readonly \
--net=host \
--user root --cap-add sys_ptrace --cap-add dac_read_search \
docker.elastic.co/beats/metricbeat:8.6.0 -e -system.hostfs=/hostfs
logs (nothing too interesting, but see for yourself):
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.070Z","log.origin":{"file.name":"instance/beat.go","file.line":724},"message":"Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.085Z","log.origin":{"file.name":"instance/beat.go","file.line":732},"message":"Beat ID: e43c03af-e0a0-4446-a351-cdd620b9976a","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.086Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.087Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1096},"message":"Beat info","service.name":"metricbeat","system_info":{"beat":{"path":{"config":"/usr/share/metricbeat","data":"/usr/share/metricbeat/data","home":"/usr/share/metricbeat","logs":"/usr/share/metricbeat/logs"},"type":"metricbeat","uuid":"e43c03af-e0a0-4446-a351-cdd620b9976a"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.087Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1105},"message":"Build info","service.name":"metricbeat","system_info":{"build":{"commit":"561a3e1839f1a50ce832e8e114de399b2bee2542","libbeat":"8.6.0","time":"2023-01-04T01:30:07.000Z","version":"8.6.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.087Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Go runtime info","service.name":"metricbeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.18.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.089Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1112},"message":"Host info","service.name":"metricbeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-01-24T09:07:03Z","containerized":false,"name":"myHost","ip":["127.0.0.1/8","::1/128","192.168.77.68/24","2003:eb:472a:ef00:299c:b7ee:d030:ceac/64","2003:eb:472a:ef00:b436:4b5e:de2c:1570/64","fe80::e84a:2745:e289:730a/64","172.17.0.1/16","fe80::42:6ff:fedb:1100/64","172.30.0.1/16","fe80::42:b3ff:fe1a:fca8/64","192.168.0.1/20","fe80::42:3bff:fe0a:89f6/64","fe80::8cc7:83ff:fe1c:dd2f/64","fe80::3871:b5ff:fec8:8c03/64"],"kernel_version":"5.15.0-58-generic","mac":["20:1a:06:f6:92:98","28:e3:47:76:0e:0b","02:42:06:db:11:00","02:42:b3:1a:fc:a8","02:42:3b:0a:89:f6","8e:c7:83:1c:dd:2f","3a:71:b5:c8:8c:03"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.5 LTS (Focal Fossa)","major":20,"minor":4,"patch":5,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.089Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1141},"message":"Process info","service.name":"metricbeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","sys_ptrace","mknod","audit_write","setfcap"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","sys_ptrace","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","sys_ptrace","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/metricbeat","exe":"/usr/share/metricbeat/metricbeat","name":"metricbeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2023-01-28T17:42:20.940Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.089Z","log.origin":{"file.name":"instance/beat.go","file.line":296},"message":"Setup Beat: metricbeat; Version: 8.6.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.103Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: http://172.17.0.1:9200","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.103Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: myHost","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-01-28T17:42:22.179Z","log.logger":"cfgwarn","log.origin":{"file.name":"sysinit/init.go","file.line":79},"message":"DEPRECATED: The --system.hostfs flag will be removed in the future and replaced by a config value. Will be removed in version: 8.0.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.186Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.186Z","log.origin":{"file.name":"instance/beat.go","file.line":486},"message":"metricbeat start running.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.189Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"config reloader started","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:22.189Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:25.072Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(http://172.17.0.1:9200))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:25.348Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.6.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:25.395Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.6.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:25.395Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":231},"message":"Auto ILM enable success.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:25.459Z","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy metricbeat exists already.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:25.460Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":366},"message":"Set settings.index.lifecycle.name in template to {metricbeat {\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_age\":\"30d\",\"max_primary_shard_size\":\"50gb\"}}}}}}} as ILM is enabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:25.530Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":115},"message":"Template \"metricbeat-8.6.0\" already exists and will not be overwritten.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:25.531Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":267},"message":"Loaded index template.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-28T17:42:25.553Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(http://172.17.0.1:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
You configuration looks ok, but you do not have almost any metricset enabled.
You have only this as the metricsets, I'm not sure that this alone will bring you anything since the process metricset is used to bring the metrics per process, but you do not any other metric.
What do you want to achieve with this config? Have you tried to add cpu, load and memory metricsets?
I was trying to make the Example as small as possible. I am getting cpu metrics to the processes. To my understanding if I take the cpu metricset I get a whole diffrent type Document (the value under event.dataset is system.cpu and with that diffrent fields).
As I already tried to explain, my problem is that I only get the processes (and metrics of those processes) of my container.
The Documents of the two Processes I get from metricbeat:
{
"@timestamp": [
"2023-01-29T14:13:28.958Z"
],
"agent.ephemeral_id": [
"fef82202-32d4-45ec-82fb-3071ea8e2ea2"
],
"agent.hostname": [
"myHost"
],
"agent.id": [
"be668fb0-2faa-4f5a-a949-aa3322da99ce"
],
"agent.name": [
"myHost"
],
"agent.type": [
"metricbeat"
],
"agent.version": [
"8.6.0"
],
"beats_state.state.host.name": [
"myHost"
],
"beats_state.timestamp": [
"2023-01-29T14:13:28.958Z"
],
"ecs.version": [
"8.0.0"
],
"event.dataset": [
"system.process"
],
"event.duration": [
182587532
],
"event.module": [
"system"
],
"host.name": [
"myHost"
],
"kibana_stats.timestamp": [
"2023-01-29T14:13:28.958Z"
],
"logstash_stats.timestamp": [
"2023-01-29T14:13:28.958Z"
],
"metricset.name": [
"process"
],
"metricset.period": [
30000
],
"mongodb.status.process": [
"tini"
],
"process.args": [
"/usr/bin/tini",
"--",
"/usr/local/bin/docker-entrypoint",
"-e",
"-system.hostfs=/hostfs"
],
"process.command_line": [
"/usr/bin/tini -- /usr/local/bin/docker-entrypoint -e -system.hostfs=/hostfs"
],
"process.command_line.text": [
"/usr/bin/tini -- /usr/local/bin/docker-entrypoint -e -system.hostfs=/hostfs"
],
"process.cpu.pct": [
0
],
"process.cpu.start_time": [
"2023-01-29T14:02:48.000Z"
],
"process.executable": [
"/usr/bin/tini"
],
"process.executable.text": [
"/usr/bin/tini"
],
"process.memory.pct": [
0
],
"process.name": [
"tini"
],
"process.name.text": [
"tini"
],
"process.parent.pid": [
1322131
],
"process.pgid": [
1322153
],
"process.pid": [
1322153
],
"process.state": [
"sleeping"
],
"process.working_directory": [
"/usr/share/metricbeat"
],
"process.working_directory.text": [
"/usr/share/metricbeat"
],
"service.type": [
"system"
],
"system.process.cgroup.cgroups_version": [
2
],
"system.process.cgroup.id": [
"/"
],
"system.process.cgroup.path": [
"/"
],
"system.process.cmdline": [
"/usr/bin/tini -- /usr/local/bin/docker-entrypoint -e -system.hostfs=/hostfs"
],
"system.process.cpu.start_time": [
"2023-01-29T14:02:48.000Z"
],
"system.process.cpu.total.norm.pct": [
0
],
"system.process.cpu.total.pct": [
0
],
"system.process.cpu.total.value": [
70
],
"system.process.fd.limit.hard": [
1048576
],
"system.process.fd.limit.soft": [
1048576
],
"system.process.fd.open": [
3
],
"system.process.memory.rss.bytes": [
602112
],
"system.process.memory.rss.pct": [
0
],
"system.process.memory.share": [
536576
],
"system.process.memory.size": [
2560000
],
"system.process.state": [
"sleeping"
],
"timestamp": [
"2023-01-29T14:13:28.958Z"
],
"user.name": [
"root"
],
"user.name.text": [
"root"
],
"_id": "by_g_YUBSiWX6G4_P1Ch",
"_index": ".ds-metricbeat-8.6.0-2023.01.25-000001",
"_score": null
}
{
"@timestamp": [
"2023-01-29T14:13:28.958Z"
],
"agent.ephemeral_id": [
"fef82202-32d4-45ec-82fb-3071ea8e2ea2"
],
"agent.hostname": [
"myHost"
],
"agent.id": [
"be668fb0-2faa-4f5a-a949-aa3322da99ce"
],
"agent.name": [
"myHost"
],
"agent.type": [
"metricbeat"
],
"agent.version": [
"8.6.0"
],
"beats_state.state.host.name": [
"myHost"
],
"beats_state.timestamp": [
"2023-01-29T14:13:28.958Z"
],
"ecs.version": [
"8.0.0"
],
"event.dataset": [
"system.process"
],
"event.duration": [
182612892
],
"event.module": [
"system"
],
"host.name": [
"myHost"
],
"kibana_stats.timestamp": [
"2023-01-29T14:13:28.958Z"
],
"logstash_stats.timestamp": [
"2023-01-29T14:13:28.958Z"
],
"metricset.name": [
"process"
],
"metricset.period": [
30000
],
"mongodb.status.process": [
"metricbeat"
],
"process.args": [
"metricbeat",
"-e",
"-system.hostfs=/hostfs"
],
"process.command_line": [
"metricbeat -e -system.hostfs=/hostfs"
],
"process.command_line.text": [
"metricbeat -e -system.hostfs=/hostfs"
],
"process.cpu.pct": [
0.002
],
"process.cpu.start_time": [
"2023-01-29T14:02:48.000Z"
],
"process.executable": [
"/usr/share/metricbeat/metricbeat"
],
"process.executable.text": [
"/usr/share/metricbeat/metricbeat"
],
"process.memory.pct": [
0.017
],
"process.name": [
"metricbeat"
],
"process.name.text": [
"metricbeat"
],
"process.parent.pid": [
1322153
],
"process.pgid": [
1322164
],
"process.pid": [
1322164
],
"process.state": [
"sleeping"
],
"process.working_directory": [
"/usr/share/metricbeat"
],
"process.working_directory.text": [
"/usr/share/metricbeat"
],
"service.type": [
"system"
],
"system.process.cgroup.cgroups_version": [
2
],
"system.process.cgroup.id": [
"/"
],
"system.process.cgroup.path": [
"/"
],
"system.process.cmdline": [
"metricbeat -e -system.hostfs=/hostfs"
],
"system.process.cpu.start_time": [
"2023-01-29T14:02:48.000Z"
],
"system.process.cpu.total.norm.pct": [
0.002
],
"system.process.cpu.total.pct": [
0.007
],
"system.process.cpu.total.value": [
5110
],
"system.process.fd.limit.hard": [
1048576
],
"system.process.fd.limit.soft": [
1048576
],
"system.process.fd.open": [
10
],
"system.process.memory.rss.bytes": [
130097152
],
"system.process.memory.rss.pct": [
0.017
],
"system.process.memory.share": [
88051712
],
"system.process.memory.size": [
1581248512
],
"system.process.state": [
"sleeping"
],
"timestamp": [
"2023-01-29T14:13:28.958Z"
],
"user.name": [
"root"
],
"user.name.text": [
"root"
],
"_id": "cC_g_YUBSiWX6G4_P1Ch",
"_index": ".ds-metricbeat-8.6.0-2023.01.25-000001",
"_score": null
}
Well, I do not see anything wrong in your configuration, it is in accord with the documentation.
If you still do not see any process from your host machine then you need to open a bug report so someone from Elastic can look on it and give some insight.
Thank you for your help.
The Link to the Github Issue.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.