# Docker and System Logs/Metrics from container-based Agent **URL:** \ **Category:** Elastic Agent\ **Created:** [January 8, 2023, 1:04pm UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679 "2023-01-08T13:04:03Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![litronics](https://avatars.discourse-cdn.com/v4/letter/l/8491ac/32.png) [@litronics](https://discuss.elastic.co/u/litronics)\ **Post date:** [January 8, 2023, 1:04pm UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/1 "2023-01-08T13:04:03Z")
Hi, I am pretty new to the ELK Stack and have some trouble getting detailed metrics from a docker based agent. The Stack is running on also as docker container and after days playing around with TLS certificates I have the Stack running. Now I am trying to deploy a dockerised agent on a raspberry pi 4 and control it via fleet. Basically that works like a charm - agent get's deployed, registers itself in fleet and receives the configuration ๐Ÿ™‚ BUT I have trouble to see all the metrics from all the docker containers on that host. When it comes to error messages, I geht the following messages over and over again: > [elastic\_agent.metricbeat][error] error getting cgroup stats for V2: error fetching stats for controller io: error fetching IO stats: error getting io.stats for path /hostfs/sys/fs/cgroup: error scanning file: /hostfs/sys/fs/cgroup/io.stat: input does not match format > [elastic\_agent.metricbeat][error] Error fetching data for metricset docker.memory: No memory stats data available I verified if the host OS is running cgroup V2 and discovered that rasbian (based on debian) is using V2. I also checked the volume mounts in my agent-container and did not find any obvious errors. Just for your information - this is my docker-compose file which I use for the agent: ```auto version: '3.5' secrets: ca.crt: file: ./cert/ca.crt services: elastic-agent: container_name: elastic-agent hostname: openhab_el-agent build: context: ./ restart: unless-stopped user: root environment: FLEET_ENROLL: 1 FLEET_ENROLLMENT_TOKEN: 'TF9#################################==' FLEET_URL: 'https://fleet-server.my.domain:8220' FLEET_CA: '/ca.crt' volumes: - /var/run/docker.sock:/var/run/docker.sock:ro - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro - /proc:/hostfs/proc:ro - /:/hostfs:ro secrets: - source: ca.crt target: /ca.crt networks: - elk networks: elk: driver: bridge ``` The Dockerfile is straight forward - just installing the root-ca certificate (otherwise the agent is not running): ```auto FROM docker.elastic.co/beats/elastic-agent:8.5.3 USER root ADD cert/ca.crt /usr/local/share/ca-certificates/ca.crt RUN chmod 644 /usr/local/share/ca-certificates/ca.crt && update-ca-certificates ``` Last thing to mention - the container runs as root and therefore I guess it should have access to all metrics within the container but also on the host OS. Any idea what's wrong with that docker configuration? ---
**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\ **Post date:** [January 9, 2023, 7:51am UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/2 "2023-01-09T07:51:55Z")
Have you provided necessary permissions to the Agent to collect the metrics of your host and other pods/ containers running on that host? I faced similar issue with metricbeat earlier and turned out I need to define some role and rolebinding objects. ---
**Author:** ![litronics](https://avatars.discourse-cdn.com/v4/letter/l/8491ac/32.png) [@litronics](https://discuss.elastic.co/u/litronics)\ **Post date:** [January 9, 2023, 12:40pm UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/3 "2023-01-09T12:40:45Z")
Yes - basically that's the reason why my container is running as root ๐Ÿ™‚ Also there are no "access denied" error messages within the logs - hence I believe that permissions are fine. Another indicator, that permissions are not the issue, is that the Docker integration is able to read out all the containers which are running on the system and delivers some metrics to the ELK Stack. However, I dont see details about each container for example. ---
**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\ **Post date:** [January 10, 2023, 1:01pm UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/4 "2023-01-10T13:01:10Z")
Hmm.. may be revisit the modules specified then and the metrics that are being captured, something like: ```auto - module: system period: 30s enabled: true metricsets: - cpu - load - memory - process_summary - network - module: kubernetes enabled: true period: 30s metricsets: - node - system - pod - containers ``` What metrics you are missing exactly ? ---
**Author:** ![litronics](https://avatars.discourse-cdn.com/v4/letter/l/8491ac/32.png) [@litronics](https://discuss.elastic.co/u/litronics)\ **Post date:** [January 10, 2023, 3:00pm UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/5 "2023-01-10T15:00:06Z")
> [@Ayush\_Mathur](#): > > What metrics you are missing exactly ? I am using the default integration settings for system and docker. On my Raspberry for example - for docker.diskio metric i am only getting the following two - docker.diskio.read.bytes - docker.diskio.write.bytes All others have the same value all the time. On another host (QNAP NAS with same agent configuration) I am only getting the following: - docker.diskio.read.bytes - docker.diskio.read.ops - docker.diskio.read.rate - docker.diskio.summary.bytes - docker.diskio.summary.ops - docker.diskio.summary.rate - docker.diskio.write.bytes - docker.diskio.write.ops - docker.diskio.write.rate As the configuration is basicalle the same for both agents I guess this is somehow related to the agent and how the agent is requesting the information from the host OS. ---
**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\ **Post date:** [January 10, 2023, 3:03pm UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/6 "2023-01-10T15:03:35Z")
Can you try configuring metricbeat to poll the metrics while keeping the Agent running. It would be interesting to compare the events of both. ---
**Author:** ![litronics](https://avatars.discourse-cdn.com/v4/letter/l/8491ac/32.png) [@litronics](https://discuss.elastic.co/u/litronics)\ **Post date:** [January 10, 2023, 3:09pm UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/7 "2023-01-10T15:09:55Z")
you mean in parallel to the agent? I thought the agent is just using metricbeat to collect the logs? or should i configura a metricbeat container next to the agent? ---
**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\ **Post date:** [January 10, 2023, 3:18pm UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/8 "2023-01-10T15:18:37Z")
Yep, metricbeat as a separate container than agent, both running on same host. We need to capture the events/ logs from both and compare to see what and where exactly is the issue. ---
**Author:** ![litronics](https://avatars.discourse-cdn.com/v4/letter/l/8491ac/32.png) [@litronics](https://discuss.elastic.co/u/litronics)\ **Post date:** [January 10, 2023, 4:24pm UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/9 "2023-01-10T16:24:22Z")
I quickly set up an other metricbeat container with the following configuration: ```auto metricbeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false processors: - add_cloud_metadata: ~ - add_docker_metadata: ~ output.elasticsearch: hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}' username: '${ELASTICSEARCH_USERNAME:}' password: '${ELASTICSEARCH_PASSWORD:}' metricbeat.modules: #-------------------------------- System Module -------------------------------- - module: system metricsets: - cpu # CPU usage - load # CPU load averages - memory # Memory usage - network # Network IO - process # Per process metrics - process_summary # Process summary - uptime # System Uptime - socket_summary # Socket summary #- core # Per CPU core usage #- diskio # Disk IO #- filesystem # File system usage for each mountpoint #- fsstat # File system summary metrics #- raid # Raid #- socket # Sockets and connection info (linux only) #- service # systemd service information enabled: true period: 10s processes: ['.*'] - module: docker metricsets: - "container" - "cpu" - "diskio" - "event" - "healthcheck" - "info" - "image" - "memory" - "network" - "network_summary" hosts: ["unix:///var/run/docker.sock"] period: 10s enabled: true ``` and getting this error message in the container logs: ```auto {"log.level":"error","@timestamp":"2023-01-10T15:41:03.689Z","log.logger":"metrics","log.origin":{"file.name":"report/report.go","file.line":234},"message":"error getting cgroup stats for V2: error fetching stats for controller io: error fetching IO stats: error getting io.stats for path /hostfs/sys/fs/cgroup: error scanning file: /hostfs/sys/fs/cgroup/io.stat: input does not match format","service.name":"metricbeat","ecs.version":"1.6.0"} ``` I am using Elasticsearch since a week or so and not 100% sure where to find everything. When I look within the discovery within the "metricbeat-\*" dataview and filter for the agent.name in question. I see metrics for the system on cpu, filesystem, memory, network, but I don't see any docker entries. Am I looking on the wrong place or is my configuration wrong? ---
**Author:** ![litronics](https://avatars.discourse-cdn.com/v4/letter/l/8491ac/32.png) [@litronics](https://discuss.elastic.co/u/litronics)\ **Post date:** [January 10, 2023, 4:57pm UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/10 "2023-01-10T16:57:15Z")
Some more information from Metricbeat ๐Ÿ™‚ I enabled some more module metrics on the system and discovered that service is not working - but that seems logical as services are not present within a container ๐Ÿ™‚ However I left all enabled (except service and raid) and ran a "metricbeat test modules" within the container: ```auto # docker exec -ti metricbeat metricbeat test modules system... cpu...OK result: { ... } load...OK result: { ... } memory...OK result: { ... } network...OK result: { ... } process... error... ERROR timeout waiting for an event process_summary...OK result: { "@timestamp": "2023-01-10T16:36:07.071Z", "event": { "dataset": "system.process.summary", "duration": 1840623, "module": "system" }, "metricset": { "name": "process_summary", "period": 10000 }, "service": { "type": "system" }, "system": { "process": { "summary": { "sleeping": 3, "total": 3 } } } } uptime...OK result: { ... } socket_summary...OK result: { ... } core...OK result: { ... } diskio...OK result: { ... } filesystem...OK result: { ... } fsstat...OK result: { ... } socket...OK result: { ... } docker... container...OK result: { "@timestamp": "2023-01-10T16:36:07.108Z", "container": { "id": "ecc5737d2190a5652557b61a6b41f0c7250ec3069d12b1dfa5830b798992e3aa", "image": { "name": "metricbeat_metricbeat" }, "name": "metricbeat", "runtime": "docker" }, "docker": { "container": { "command": "/usr/bin/tini -- /usr/local/bin/docker-entrypoint -environment container", "created": "2023-01-10T15:36:28.000Z", "ip_addresses": [ "172.21.0.2" ], "labels": { "com_docker_compose_config-hash": "b3649497f15da387a04a0ba60abc6dda1b906e54b14cf1483e0026e36500a512", "com_docker_compose_container-number": "1", "com_docker_compose_oneoff": "False", "com_docker_compose_project": "metricbeat", "com_docker_compose_project_config_files": "docker-compose.yml", "com_docker_compose_project_working_dir": "/home/pi/metricbeat", "com_docker_compose_service": "metricbeat", "com_docker_compose_version": "1.25.0", "description": "Metricbeat is a lightweight shipper for metrics.", "io_k8s_description": "Metricbeat is a lightweight shipper for metrics.", "io_k8s_display-name": "Metricbeat image", "license": "Elastic License", "maintainer": "infra@elastic.co", "name": "metricbeat", "org_label-schema_build-date": "2022-12-04T04:51:03Z", "org_label-schema_license": "Elastic License", "org_label-schema_name": "metricbeat", "org_label-schema_schema-version": "1.0", "org_label-schema_url": "https://www.elastic.co/beats/metricbeat", "org_label-schema_vcs-ref": "6d03209df870c63ef9d59d609268c11dfdc835dd", "org_label-schema_vcs-url": "github.com/elastic/beats/v7", "org_label-schema_vendor": "Elastic", "org_label-schema_version": "8.5.3", "org_opencontainers_image_created": "2022-12-04T04:51:03Z", "org_opencontainers_image_licenses": "Elastic License", "org_opencontainers_image_title": "Metricbeat", "org_opencontainers_image_vendor": "Elastic", "release": "1", "summary": "metricbeat", "url": "https://www.elastic.co/beats/metricbeat", "vendor": "Elastic", "version": "8.5.3" }, "size": { "root_fs": 0, "rw": 0 }, "status": "Up 59 minutes" } }, "event": { "dataset": "docker.container", "duration": 13095583, "module": "docker" }, "metricset": { "name": "container", "period": 10000 }, "service": { "address": "unix:///var/run/docker.sock", "type": "docker" } } cpu...OK result: { "@timestamp": "2023-01-10T16:36:07.123Z", "container": { "cpu": { "usage": 0.0013759124087591242 }, "id": "9d41cbfce6ea05fa2192c9fce8f1db33339390f763e58ab798cb43572ae9af9e", "image": { "name": "elastic-agent_elastic-agent" }, "name": "elastic-agent", "runtime": "docker" }, "docker": { "container": { "labels": { "com_docker_compose_config-hash": "63e4287ebcbe6ac2019cfaf831ff5bf9ab087163358b51d4712c9de468d3f174", "com_docker_compose_container-number": "1", "com_docker_compose_oneoff": "False", "com_docker_compose_project": "elastic-agent", "com_docker_compose_project_config_files": "docker-compose.yml", "com_docker_compose_project_working_dir": "/home/pi/elastic-agent", "com_docker_compose_service": "elastic-agent", "com_docker_compose_version": "1.25.0", "description": "Agent manages other beats based on configuration provided.", "io_k8s_description": "Agent manages other beats based on configuration provided.", "io_k8s_display-name": "Elastic-Agent image", "license": "Elastic License", "maintainer": "infra@elastic.co", "name": "elastic-agent", "org_label-schema_build-date": "2022-12-06T00:06:31Z", "org_label-schema_license": "Elastic License", "org_label-schema_name": "elastic-agent", "org_label-schema_schema-version": "1.0", "org_label-schema_url": "https://www.elastic.co/beats/elastic-agent", "org_label-schema_vcs-ref": "0e1a7396f356f8af5ad28c3c8e941256f50c86a2", "org_label-schema_vcs-url": "github.com/elastic/elastic-agent", "org_label-schema_vendor": "Elastic", "org_label-schema_version": "8.5.3", "org_opencontainers_image_created": "2022-12-06T00:06:31Z", "org_opencontainers_image_licenses": "Elastic License", "org_opencontainers_image_title": "Elastic-Agent", "org_opencontainers_image_vendor": "Elastic", "release": "1", "summary": "elastic-agent", "url": "https://www.elastic.co/beats/elastic-agent", "vendor": "Elastic", "version": "8.5.3" } }, "cpu": { "core": {}, "kernel": { "norm": { "pct": 0.0001411192214111922 }, "pct": 0.0005644768856447688, "ticks": 67300603000 }, "system": { "norm": { "pct": 1 }, "pct": 4, "ticks": 30896080000000 }, "total": { "norm": { "pct": 0.0013759124087591242 }, "pct": 0.005503649635036497 }, "user": { "norm": { "pct": 0.0012345498783454987 }, "pct": 0.004938199513381995, "ticks": 95724911000 } } }, "event": { "dataset": "docker.cpu", "duration": 2139403607, "module": "docker" }, "metricset": { "name": "cpu", "period": 10000 }, "service": { "address": "unix:///var/run/docker.sock", "type": "docker" } } diskio...OK result: { "@timestamp": "2023-01-10T16:36:09.269Z", "container": { "disk": { "read": { "bytes": 176128 }, "write": { "bytes": 122880 } }, "id": "ecc5737d2190a5652557b61a6b41f0c7250ec3069d12b1dfa5830b798992e3aa", "image": { "name": "metricbeat_metricbeat" }, "name": "metricbeat", "runtime": "docker" }, "docker": { "container": { "labels": { "com_docker_compose_config-hash": "b3649497f15da387a04a0ba60abc6dda1b906e54b14cf1483e0026e36500a512", "com_docker_compose_container-number": "1", "com_docker_compose_oneoff": "False", "com_docker_compose_project": "metricbeat", "com_docker_compose_project_config_files": "docker-compose.yml", "com_docker_compose_project_working_dir": "/home/pi/metricbeat", "com_docker_compose_service": "metricbeat", "com_docker_compose_version": "1.25.0", "description": "Metricbeat is a lightweight shipper for metrics.", "io_k8s_description": "Metricbeat is a lightweight shipper for metrics.", "io_k8s_display-name": "Metricbeat image", "license": "Elastic License", "maintainer": "infra@elastic.co", "name": "metricbeat", "org_label-schema_build-date": "2022-12-04T04:51:03Z", "org_label-schema_license": "Elastic License", "org_label-schema_name": "metricbeat", "org_label-schema_schema-version": "1.0", "org_label-schema_url": "https://www.elastic.co/beats/metricbeat", "org_label-schema_vcs-ref": "6d03209df870c63ef9d59d609268c11dfdc835dd", "org_label-schema_vcs-url": "github.com/elastic/beats/v7", "org_label-schema_vendor": "Elastic", "org_label-schema_version": "8.5.3", "org_opencontainers_image_created": "2022-12-04T04:51:03Z", "org_opencontainers_image_licenses": "Elastic License", "org_opencontainers_image_title": "Metricbeat", "org_opencontainers_image_vendor": "Elastic", "release": "1", "summary": "metricbeat", "url": "https://www.elastic.co/beats/metricbeat", "vendor": "Elastic", "version": "8.5.3" } }, "diskio": { "read": { "bytes": 176128, "ops": 0, "queued": 0, "rate": 0, "service_time": 0, "wait_time": 0 }, "summary": { "bytes": 0, "ops": 0, "queued": 0, "rate": 0, "service_time": 0, "wait_time": 0 }, "write": { "bytes": 122880, "ops": 0, "queued": 0, "rate": 0, "service_time": 0, "wait_time": 0 } } }, "event": { "dataset": "docker.diskio", "duration": 2133323204, "module": "docker" }, "metricset": { "name": "diskio", "period": 10000 }, "service": { "address": "unix:///var/run/docker.sock", "type": "docker" } } event... error... ERROR timeout waiting for an event healthcheck...OK result: { ... } info...OK result: { ... } image...OK result: { ... } memory...OK result: { ... } network...OK result: { ... } network_summary... error... ERROR error fetching namespace for PID 1424: error reading network namespace link: readlink /proc/1424/ns/net: no such file or directory system... cpu...OK result: { ... } load...OK result: { ... } memory...OK result: { ... } network...OK result: { ... } process... error... ERROR timeout waiting for an event process_summary...OK result: { ... } socket_summary...OK result: { ... } system... filesystem...OK result: { ... } fsstat...OK result: { ... } system... uptime...OK result: { ... } ``` I deleted most of the results but left all the errors and the _ **diskio...** _ results - as there are some metrics (diskio, summary, write) which are defenetly wrong. Hope that helps somehow to identify the root cause? ---
**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\ **Post date:** [January 11, 2023, 9:57am UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/11 "2023-01-11T09:57:22Z")
When you run the metricbeat container, are you specifying this as argument to container: ```auto spec: template: spec: containers: - args: [ "-e", "-system.hostfs=/hostfs" ] "securityContext": "runAsUser": 0 ``` Also you need to mount volume `/proc` and `/sys/fs/cgroup` to your container. See: [Kubespray-4RPI4/09-metricbeat.yml at main ยท bpcvdhelm/Kubespray-4RPI4 (github.com)](https://github.com/bpcvdhelm/Kubespray-4RPI4/blob/main/09-metricbeat.yml) for metricbeat configuration on Raspberry PI. ---
**Author:** ![litronics](https://avatars.discourse-cdn.com/v4/letter/l/8491ac/32.png) [@litronics](https://discuss.elastic.co/u/litronics)\ **Post date:** [January 12, 2023, 9:19am UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/12 "2023-01-12T09:19:09Z")
> [@Ayush\_Mathur](#): > > Also you need to mount volume `/proc` and `/sys/fs/cgroup` to your container. See: [Kubespray-4RPI4/09-metricbeat.yml at main ยท bpcvdhelm/Kubespray-4RPI4 (github.com)](https://github.com/bpcvdhelm/Kubespray-4RPI4/blob/main/09-metricbeat.yml) for metricbeat configuration on Raspberry PI. Yes I mounted those filesystems within the container. This is my docker-compose.yml: ```auto version: '3.5' secrets: ca.crt: file: ./cert/ca.crt services: metricbeat: container_name: metricbeat hostname: openhab_metricbeat build: context: ./ restart: unless-stopped user: root command: - -e - --strict.perms=false environment: ELASTICSEARCH_CA: '/ca.crt' ELASTICSEARCH_HOSTS: 'https://elasticsearch.my.domain:9200' ELASTICSEARCH_USERNAME: ' ****************' ELASTICSEARCH_PASSWORD: ' ****************' KIBANA_HOST: 'https://kibana.my.domain:5601' volumes: - ./config/metricbeat.docker.yml:/usr/share/metricbeat/metricbeat.yml:ro - /var/run/docker.sock:/var/run/docker.sock:ro - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro - /proc:/hostfs/proc:ro - /:/hostfs:ro secrets: - source: ca.crt target: /ca.crt networks: - elk networks: elk: driver: bridge ``` Passing the "-system.hostfs=/hostfs" as command just produces an errormessage telling me that this is not supported anymore and I supposed to use the config file. Hence I updated the config file which looks like this: ```auto metricbeat.config.modules: path: ${path.config}/modules.d/*.yml path: ${system.hostfs}/hostfs reload.enabled: false processors: - add_cloud_metadata: ~ - add_docker_metadata: ~ output.elasticsearch: hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}' username: '${ELASTICSEARCH_USERNAME:}' password: '${ELASTICSEARCH_PASSWORD:}' metricbeat.modules: #-------------------------------- System Module -------------------------------- - module: system metricsets: - cpu # CPU usage - load # CPU load averages - memory # Memory usage - network # Network IO - process # Per process metrics - process_summary # Process summary - uptime # System Uptime - socket_summary # Socket summary - core # Per CPU core usage - diskio # Disk IO - filesystem # File system usage for each mountpoint - fsstat # File system summary metrics #- raid # Raid - socket # Sockets and connection info (linux only) #- service # systemd service information enabled: true period: 10s processes: ['.*'] - module: docker metricsets: - "container" - "cpu" - "diskio" - "event" - "healthcheck" - "info" - "image" - "memory" - "network" - "network_summary" hosts: ["unix:///var/run/docker.sock"] period: 10s enabled: true # ================================== General =================================== tags: ["openhab", "docker"] ``` With these updates I am still don't get the details from disk-io and in the logs I find this error messages: ```auto {"log.level":"error","@timestamp":"2023-01-12T09:07:00.197Z","log.origin":{"file.name":"module/wrapper.go","file.line":263},"message":"Error fetching data for metricset docker.network_summary: error fetching namespace for PID 1206: error reading network namespace link: readlink /proc/1206/ns/net: no such file or directory","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2023-01-12T09:07:53.385Z","log.logger":"metrics","log.origin":{"file.name":"report/report.go","file.line":234},"message":"error getting cgroup stats for V2: error fetching stats for controller io: error fetching IO stats: error getting io.stats for path /hostfs/sys/fs/cgroup: error scanning file: /hostfs/sys/fs/cgroup/io.stat: input does not match format","service.name":"metricbeat","ecs.version":"1.6.0"} ``` There are also a lots of warnings: ```auto {"log.level":"warn","@timestamp":"2023-01-12T09:10:10.423Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 10, 7, 61010011, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"614c6767-850e-4f8e-b208-b3adfaabe295\",\"id\":\"d7c7dfa6-025a-4d96-b8f6-ae5fa6362aec\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"container\":{\"id\":\"cf2425a968152ad0d8a119da720f777fb69b70bc50574f8ac139ce483fe830e7\",\"image\":{\"name\":\"ghcr.io/jens-maus/raspberrymatic:latest\"},\"memory\":{\"usage\":0.13816561590189386},\"name\":\"ccu\",\"runtime\":\"docker\"},\"docker\":{\"container\":{\"labels\":{\"com_datadoghq_ad_logs\":\"[{\\\"source\\\": \\\"raspberrymatic\\\", \\\"service\\\": \\\"raspberrymatic\\\"}]\",\"com_docker_compose_config-hash\":\"a987e3a8d753f42ddb1cdb844ced8e93d734c4f4b64d111b2b67e0012ab46779\",\"com_docker_compose_container-number\":\"1\",\"com_docker_compose_oneoff\":\"False\",\"com_docker_compose_project\":\"docker\",\"com_docker_compose_project_config_files\":\"docker-compose.yaml\",\"com_docker_compose_project_working_dir\":\"/home/pi/docker\",\"com_docker_compose_service\":\"raspberrymatic\",\"com_docker_compose_version\":\"1.25.0\",\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"}},\"memory\":{\"fail\":{\"count\":0},\"limit\":3978674176,\"rss\":{\"pct\":0,\"total\":0},\"stats\":{\"active_anon\":9695232,\"active_file\":105422848,\"anon\":409272320,\"anon_thp\":0,\"file\":153096192,\"file_dirty\":0,\"file_mapped\":25309184,\"file_writeback\":0,\"inactive_anon\":409288704,\"inactive_file\":37974016,\"kernel_stack\":2015232,\"pgactivate\":38042,\"pgdeactivate\":10736,\"pgfault\":23461057,\"pglazyfree\":0,\"pglazyfreed\":0,\"pgmajfault\":530,\"pgrefill\":15905,\"pgscan\":117876,\"pgsteal\":33783,\"shmem\":9699328,\"slab\":21011752,\"slab_reclaimable\":18228888,\"slab_unreclaimable\":2782864,\"sock\":36864,\"thp_collapse_alloc\":0,\"thp_fault_alloc\":0,\"unevictable\":0,\"workingset_activate\":0,\"workingset_nodereclaim\":0,\"workingset_refault\":0},\"usage\":{\"max\":0,\"pct\":0.13816561590189386,\"total\":549715968}}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.memory\",\"duration\":2109356618,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"memory\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.container.labels.org_opencontainers_image_created] of type [date] in document with id 'WXI-pYUBBZFRz5sZbkmE'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-01-12T09:10:10.423Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 10, 9, 837451224, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"614c6767-850e-4f8e-b208-b3adfaabe295\",\"id\":\"d7c7dfa6-025a-4d96-b8f6-ae5fa6362aec\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"docker\":{\"image\":{\"created\":\"2022-12-18T09:09:28.000Z\",\"id\":{\"current\":\"sha256:3b270c8f016295f86115f5e6c340deb4f225a5270858935220364c7889c4edfb\",\"parent\":\"\"},\"labels\":{\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"},\"size\":{\"regular\":488336932,\"virtual\":488336932},\"tags\":[\"ghcr.io/jens-maus/raspberrymatic:latest\"]}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.image\",\"duration\":82741571,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"image\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.image.labels.org_opencontainers_image_created] of type [date] in document with id 'YnI-pYUBBZFRz5sZbkmE'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-01-12T09:10:12.915Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 10, 9, 249664408, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"614c6767-850e-4f8e-b208-b3adfaabe295\",\"id\":\"d7c7dfa6-025a-4d96-b8f6-ae5fa6362aec\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"container\":{\"disk\":{\"read\":{\"bytes\":236494848},\"write\":{\"bytes\":64557056}},\"id\":\"cf2425a968152ad0d8a119da720f777fb69b70bc50574f8ac139ce483fe830e7\",\"image\":{\"name\":\"ghcr.io/jens-maus/raspberrymatic:latest\"},\"name\":\"ccu\",\"runtime\":\"docker\"},\"docker\":{\"container\":{\"labels\":{\"com_datadoghq_ad_logs\":\"[{\\\"source\\\": \\\"raspberrymatic\\\", \\\"service\\\": \\\"raspberrymatic\\\"}]\",\"com_docker_compose_config-hash\":\"a987e3a8d753f42ddb1cdb844ced8e93d734c4f4b64d111b2b67e0012ab46779\",\"com_docker_compose_container-number\":\"1\",\"com_docker_compose_oneoff\":\"False\",\"com_docker_compose_project\":\"docker\",\"com_docker_compose_project_config_files\":\"docker-compose.yaml\",\"com_docker_compose_project_working_dir\":\"/home/pi/docker\",\"com_docker_compose_service\":\"raspberrymatic\",\"com_docker_compose_version\":\"1.25.0\",\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"}},\"diskio\":{\"read\":{\"bytes\":236494848,\"ops\":0,\"queued\":0,\"rate\":0,\"service_time\":0,\"wait_time\":0},\"summary\":{\"bytes\":0,\"ops\":0,\"queued\":0,\"rate\":0,\"service_time\":0,\"wait_time\":0},\"write\":{\"bytes\":64557056,\"ops\":0,\"queued\":0,\"rate\":0,\"service_time\":0,\"wait_time\":0}}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.diskio\",\"duration\":2014061721,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"diskio\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.container.labels.org_opencontainers_image_created] of type [date] in document with id 'fnI-pYUBBZFRz5sZdkqu'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-01-12T09:10:14.907Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 10, 11, 349918713, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"614c6767-850e-4f8e-b208-b3adfaabe295\",\"id\":\"d7c7dfa6-025a-4d96-b8f6-ae5fa6362aec\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"container\":{\"id\":\"cf2425a968152ad0d8a119da720f777fb69b70bc50574f8ac139ce483fe830e7\",\"image\":{\"name\":\"ghcr.io/jens-maus/raspberrymatic:latest\"},\"name\":\"ccu\",\"network\":{\"egress\":{\"bytes\":80924287},\"ingress\":{\"bytes\":34736784}},\"runtime\":\"docker\"},\"docker\":{\"container\":{\"labels\":{\"com_datadoghq_ad_logs\":\"[{\\\"source\\\": \\\"raspberrymatic\\\", \\\"service\\\": \\\"raspberrymatic\\\"}]\",\"com_docker_compose_config-hash\":\"a987e3a8d753f42ddb1cdb844ced8e93d734c4f4b64d111b2b67e0012ab46779\",\"com_docker_compose_container-number\":\"1\",\"com_docker_compose_oneoff\":\"False\",\"com_docker_compose_project\":\"docker\",\"com_docker_compose_project_config_files\":\"docker-compose.yaml\",\"com_docker_compose_project_working_dir\":\"/home/pi/docker\",\"com_docker_compose_service\":\"raspberrymatic\",\"com_docker_compose_version\":\"1.25.0\",\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"}},\"network\":{\"inbound\":{\"bytes\":34736784,\"dropped\":0,\"errors\":0,\"packets\":171201},\"interface\":\"eth0\",\"outbound\":{\"bytes\":80924287,\"dropped\":0,\"errors\":0,\"packets\":266506}}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.network\",\"duration\":2014493548,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"network\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.container.labels.org_opencontainers_image_created] of type [date] in document with id 'MnI-pYUBBZFRz5sZf0sf'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-01-12T09:10:17.905Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 10, 16, 533410422, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"614c6767-850e-4f8e-b208-b3adfaabe295\",\"id\":\"d7c7dfa6-025a-4d96-b8f6-ae5fa6362aec\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"container\":{\"id\":\"cf2425a968152ad0d8a119da720f777fb69b70bc50574f8ac139ce483fe830e7\",\"image\":{\"name\":\"ghcr.io/jens-maus/raspberrymatic:latest\"},\"name\":\"ccu\",\"runtime\":\"docker\"},\"docker\":{\"container\":{\"command\":\"/sbin/init\",\"created\":\"2023-01-02T10:00:25.000Z\",\"ip_addresses\":[\"172.18.0.2\"],\"labels\":{\"com_datadoghq_ad_logs\":\"[{\\\"source\\\": \\\"raspberrymatic\\\", \\\"service\\\": \\\"raspberrymatic\\\"}]\",\"com_docker_compose_config-hash\":\"a987e3a8d753f42ddb1cdb844ced8e93d734c4f4b64d111b2b67e0012ab46779\",\"com_docker_compose_container-number\":\"1\",\"com_docker_compose_oneoff\":\"False\",\"com_docker_compose_project\":\"docker\",\"com_docker_compose_project_config_files\":\"docker-compose.yaml\",\"com_docker_compose_project_working_dir\":\"/home/pi/docker\",\"com_docker_compose_service\":\"raspberrymatic\",\"com_docker_compose_version\":\"1.25.0\",\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"},\"size\":{\"root_fs\":0,\"rw\":0},\"status\":\"Up 43 hours (healthy)\"}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.container\",\"duration\":11304304,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"container\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.container.labels.org_opencontainers_image_created] of type [date] in document with id 'QXI-pYUBBZFRz5sZikxx'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"} { ``` it look like that those mainly related to the docker module. ---
**Author:** ![litronics](https://avatars.discourse-cdn.com/v4/letter/l/8491ac/32.png) [@litronics](https://discuss.elastic.co/u/litronics)\ **Post date:** [January 12, 2023, 9:26am UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/13 "2023-01-12T09:26:32Z")
I thought it might be helpful to see the startup log of the container and therefor deleted the container and captured the logs right from the beginning: ```auto {"log.level":"info","@timestamp":"2023-01-12T09:20:36.306Z","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:36.329Z","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: a30ffb34-5553-404e-ab63-c61016f74379","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-01-12T09:20:39.331Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.374Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.375Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1082},"message":"Beat info","service.name":"metricbeat","system_info":{"beat":{"path":{"config":"/usr/share/metricbeat","data":"/usr/share/metricbeat/data","home":"/usr/share/metricbeat","logs":"/usr/share/metricbeat/logs"},"type":"metricbeat","uuid":"a30ffb34-5553-404e-ab63-c61016f74379"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.375Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1091},"message":"Build info","service.name":"metricbeat","system_info":{"build":{"commit":"6d03209df870c63ef9d59d609268c11dfdc835dd","libbeat":"8.5.3","time":"2022-12-04T04:51:35.000Z","version":"8.5.3"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.375Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1094},"message":"Go runtime info","service.name":"metricbeat","system_info":{"go":{"os":"linux","arch":"arm64","max_procs":4,"version":"go1.18.7"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.378Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1098},"message":"Host info","service.name":"metricbeat","system_info":{"host":{"architecture":"aarch64","boot_time":"2023-01-10T14:26:35Z","containerized":false,"name":"openhab_metricbeat","ip":["127.0.0.1/8","172.21.0.2/16"],"kernel_version":"5.15.84-v8+","mac":["02:42:ac:15:00:02"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.5 LTS (Focal Fossa)","major":20,"minor":4,"patch":5,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.379Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1127},"message":"Process info","service.name":"metricbeat","system_info":{"process":{"capabilities":{"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/metricbeat","exe":"/usr/share/metricbeat/metricbeat","name":"metricbeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2023-01-12T09:20:35.400Z"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.380Z","log.origin":{"file.name":"instance/beat.go","file.line":294},"message":"Setup Beat: metricbeat; Version: 8.5.3","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.387Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://elasticsearch.my.domain:9200","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.388Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: openhab_metricbeat","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.411Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":102},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.476Z","log.origin":{"file.name":"filesystem/filesystem.go","file.line":70},"message":"Ignoring filesystem types: sysfs, tmpfs, bdev, proc, cgroup, cgroup2, cpuset, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, bpf, pipefs, ramfs, rpc_pipefs, devpts, nfs, nfs4, autofs, mqueue, binder, pstore, fuse, fusectl, overlay","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.477Z","log.logger":"system.fsstat","log.origin":{"file.name":"fsstat/fsstat.go","file.line":60},"message":"Ignoring filesystem types: %ssysfs, tmpfs, bdev, proc, cgroup, cgroup2, cpuset, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, bpf, pipefs, ramfs, rpc_pipefs, devpts, nfs, nfs4, autofs, mqueue, binder, pstore, fuse, fusectl, overlay","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.480Z","log.origin":{"file.name":"socket/socket.go","file.line":83},"message":"socket process info will only be available for processes owned by the 0 user because this Beat is not running with enough privileges","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-01-12T09:20:39.484Z","log.logger":"cfgwarn","log.origin":{"file.name":"network_summary/network_summary.go","file.line":58},"message":"BETA: The docker network_summary metricset is beta.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.486Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Starting metrics logging every 30s","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.486Z","log.origin":{"file.name":"instance/beat.go","file.line":471},"message":"metricbeat start running.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.487Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"Config reloader started","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:39.488Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:40.581Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(https://elasticsearch.my.domain:9200))","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:40.661Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.5.3","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:40.674Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.5.3","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:40.674Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":231},"message":"Auto ILM enable success.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:40.685Z","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy metricbeat exists already.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:40.686Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":366},"message":"Set settings.index.lifecycle.name in template to {metricbeat {\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_age\":\"30d\",\"max_primary_shard_size\":\"50gb\"}}}}}}} as ILM is enabled.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:40.701Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":115},"message":"Template \"metricbeat-8.5.3\" already exists and will not be overwritten.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:40.701Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":267},"message":"Loaded index template.","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-01-12T09:20:40.704Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(https://elasticsearch.my.domain:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2023-01-12T09:20:46.019Z","log.origin":{"file.name":"module/wrapper.go","file.line":263},"message":"Error fetching data for metricset docker.network_summary: error fetching namespace for PID 7956: error reading network namespace link: readlink /proc/7956/ns/net: no such file or directory","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-01-12T09:20:52.504Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 20, 40, 835834909, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"de227a3d-f661-4565-8f3d-8c1392bebc8e\",\"id\":\"a30ffb34-5553-404e-ab63-c61016f74379\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"container\":{\"id\":\"cf2425a968152ad0d8a119da720f777fb69b70bc50574f8ac139ce483fe830e7\",\"image\":{\"name\":\"ghcr.io/jens-maus/raspberrymatic:latest\"},\"name\":\"ccu\",\"runtime\":\"docker\"},\"docker\":{\"container\":{\"command\":\"/sbin/init\",\"created\":\"2023-01-02T10:00:25.000Z\",\"ip_addresses\":[\"172.18.0.2\"],\"labels\":{\"com_datadoghq_ad_logs\":\"[{\\\"source\\\": \\\"raspberrymatic\\\", \\\"service\\\": \\\"raspberrymatic\\\"}]\",\"com_docker_compose_config-hash\":\"a987e3a8d753f42ddb1cdb844ced8e93d734c4f4b64d111b2b67e0012ab46779\",\"com_docker_compose_container-number\":\"1\",\"com_docker_compose_oneoff\":\"False\",\"com_docker_compose_project\":\"docker\",\"com_docker_compose_project_config_files\":\"docker-compose.yaml\",\"com_docker_compose_project_working_dir\":\"/home/pi/docker\",\"com_docker_compose_service\":\"raspberrymatic\",\"com_docker_compose_version\":\"1.25.0\",\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"},\"size\":{\"root_fs\":0,\"rw\":0},\"status\":\"Up 43 hours (healthy)\"}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.container\",\"duration\":13389279,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"container\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.container.labels.org_opencontainers_image_created] of type [date] in document with id 'p3JIpYUBBZFRz5sZOb4l'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"} ``` ---
**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\ **Post date:** [January 12, 2023, 10:27am UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/14 "2023-01-12T10:27:52Z")
> [@litronics](#): > > `failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]` Since this format is not supported implicitly by ES, guess you need to define your own format in your index template to avoid ingestion failure issues: [format | Elasticsearch Guide [8.6] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html#mapping-date-format) About the other issue where metrics are not being polled, guess there was another issue opened without any responses: [Metricbeat error about scanning empty io.stat file - Elastic Stack / Beats - Discuss the Elastic Stack](https://discuss.elastic.co/t/metricbeat-error-about-scanning-empty-io-stat-file/308063) And I see you have already commented on github issue for this ๐Ÿ™‚ ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 9, 2023, 10:28am UTC](https://discuss.elastic.co/t/docker-and-system-logs-metrics-from-container-based-agent/322679/15 "2023-02-09T10:28:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.