Docker and System Logs/Metrics from container-based Agent

Elastic Stack Elastic Agent
1

Hi,
I am pretty new to the ELK Stack and have some trouble getting detailed metrics from a docker based agent.

The Stack is running on also as docker container and after days playing around with TLS certificates I have the Stack running.

Now I am trying to deploy a dockerised agent on a raspberry pi 4 and control it via fleet.

Basically that works like a charm - agent get's deployed, registers itself in fleet and receives the configuration :slight_smile:
BUT I have trouble to see all the metrics from all the docker containers on that host.

When it comes to error messages, I geht the following messages over and over again:

[elastic_agent.metricbeat][error] error getting cgroup stats for V2: error fetching stats for controller io: error fetching IO stats: error getting io.stats for path /hostfs/sys/fs/cgroup: error scanning file: /hostfs/sys/fs/cgroup/io.stat: input does not match format

[elastic_agent.metricbeat][error] Error fetching data for metricset docker.memory: No memory stats data available

I verified if the host OS is running cgroup V2 and discovered that rasbian (based on debian) is using V2.
I also checked the volume mounts in my agent-container and did not find any obvious errors.

Just for your information - this is my docker-compose file which I use for the agent:

version: '3.5'

secrets:
  ca.crt:
    file: ./cert/ca.crt

services:
  elastic-agent:
    container_name: elastic-agent
    hostname: openhab_el-agent
    build:
      context: ./
    restart: unless-stopped
    user: root
    environment:
      FLEET_ENROLL: 1
      FLEET_ENROLLMENT_TOKEN: 'TF9#################################=='
      FLEET_URL: 'https://fleet-server.my.domain:8220'
      FLEET_CA: '/ca.crt'

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /proc:/hostfs/proc:ro
      - /:/hostfs:ro
    secrets:
      - source: ca.crt
        target: /ca.crt
    networks:
      - elk


networks:
  elk:
    driver: bridge

The Dockerfile is straight forward - just installing the root-ca certificate (otherwise the agent is not running):

FROM docker.elastic.co/beats/elastic-agent:8.5.3

USER root

ADD cert/ca.crt /usr/local/share/ca-certificates/ca.crt
RUN chmod 644 /usr/local/share/ca-certificates/ca.crt && update-ca-certificates

Last thing to mention - the container runs as root and therefore I guess it should have access to all metrics within the container but also on the host OS.

Any idea what's wrong with that docker configuration?

2

Have you provided necessary permissions to the Agent to collect the metrics of your host and other pods/ containers running on that host? I faced similar issue with metricbeat earlier and turned out I need to define some role and rolebinding objects.

3

Yes - basically that's the reason why my container is running as root :slight_smile:

Also there are no "access denied" error messages within the logs - hence I believe that permissions are fine.

Another indicator, that permissions are not the issue, is that the Docker integration is able to read out all the containers which are running on the system and delivers some metrics to the ELK Stack. However, I dont see details about each container for example.

4

Hmm.. may be revisit the modules specified then and the metrics that are being captured, something like:

- module: system
  period: 30s
  enabled: true
  metricsets:
    - cpu
    - load
    - memory
    - process_summary
    - network

- module: kubernetes
  enabled: true
  period: 30s
  metricsets:
    - node
    - system
    - pod
    - containers

What metrics you are missing exactly ?

5

I am using the default integration settings for system and docker.

On my Raspberry for example - for docker.diskio metric i am only getting the following two

  • docker.diskio.read.bytes
  • docker.diskio.write.bytes

All others have the same value all the time.

On another host (QNAP NAS with same agent configuration) I am only getting the following:

  • docker.diskio.read.bytes
  • docker.diskio.read.ops
  • docker.diskio.read.rate
  • docker.diskio.summary.bytes
  • docker.diskio.summary.ops
  • docker.diskio.summary.rate
  • docker.diskio.write.bytes
  • docker.diskio.write.ops
  • docker.diskio.write.rate

As the configuration is basicalle the same for both agents I guess this is somehow related to the agent and how the agent is requesting the information from the host OS.

6

Can you try configuring metricbeat to poll the metrics while keeping the Agent running. It would be interesting to compare the events of both.

7

you mean in parallel to the agent?

I thought the agent is just using metricbeat to collect the logs?

or should i configura a metricbeat container next to the agent?

8

Yep, metricbeat as a separate container than agent, both running on same host. We need to capture the events/ logs from both and compare to see what and where exactly is the issue.

9

I quickly set up an other metricbeat container with the following configuration:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

metricbeat.modules:

#-------------------------------- System Module --------------------------------
- module: system
  metricsets:
    - cpu             # CPU usage
    - load            # CPU load averages
    - memory          # Memory usage
    - network         # Network IO
    - process         # Per process metrics
    - process_summary # Process summary
    - uptime          # System Uptime
    - socket_summary  # Socket summary
    #- core           # Per CPU core usage
    #- diskio         # Disk IO
    #- filesystem     # File system usage for each mountpoint
    #- fsstat         # File system summary metrics
    #- raid           # Raid
    #- socket         # Sockets and connection info (linux only)
    #- service        # systemd service information
  enabled: true
  period: 10s
  processes: ['.*']

- module: docker
  metricsets:
    - "container"
    - "cpu"
    - "diskio"
    - "event"
    - "healthcheck"
    - "info"
    - "image"
    - "memory"
    - "network"
    - "network_summary"
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true

and getting this error message in the container logs:

{"log.level":"error","@timestamp":"2023-01-10T15:41:03.689Z","log.logger":"metrics","log.origin":{"file.name":"report/report.go","file.line":234},"message":"error getting cgroup stats for V2: error fetching stats for controller io: error fetching IO stats: error getting io.stats for path /hostfs/sys/fs/cgroup: error scanning file: /hostfs/sys/fs/cgroup/io.stat: input does not match format","service.name":"metricbeat","ecs.version":"1.6.0"}

I am using Elasticsearch since a week or so and not 100% sure where to find everything.

When I look within the discovery within the "metricbeat-*" dataview and filter for the agent.name in question. I see metrics for the system on cpu, filesystem, memory, network, but I don't see any docker entries.
Am I looking on the wrong place or is my configuration wrong?

10

Some more information from Metricbeat :slight_smile:
I enabled some more module metrics on the system and discovered that service is not working - but that seems logical as services are not present within a container :slight_smile:

However I left all enabled (except service and raid) and ran a "metricbeat test modules" within the container:

# docker exec -ti metricbeat metricbeat test modules
system...
  cpu...OK
    result:
    {
	   ...
    }

  load...OK
    result:
    {
	   ...
    }

  memory...OK
    result:
    {
       ...
    }

  network...OK
    result:
    {
        ...
    }

  process...
    error... ERROR timeout waiting for an event
  process_summary...OK
    result:
    {
     "@timestamp": "2023-01-10T16:36:07.071Z",
     "event": {
      "dataset": "system.process.summary",
      "duration": 1840623,
      "module": "system"
     },
     "metricset": {
      "name": "process_summary",
      "period": 10000
     },
     "service": {
      "type": "system"
     },
     "system": {
      "process": {
       "summary": {
        "sleeping": 3,
        "total": 3
       }
      }
     }
    }

  uptime...OK
    result:
    {
        ...
    }

  socket_summary...OK
    result:
    {
        ...
    }

  core...OK
    result:
    {
        ...
    }

  diskio...OK
    result:
    {
        ...
    }

  filesystem...OK
    result:
    {
        ...
    }

  fsstat...OK
    result:
    {
        ...
    }

  socket...OK
    result:
    {
        ...
    }

docker...
  container...OK
    result:
    {
     "@timestamp": "2023-01-10T16:36:07.108Z",
     "container": {
      "id": "ecc5737d2190a5652557b61a6b41f0c7250ec3069d12b1dfa5830b798992e3aa",
      "image": {
       "name": "metricbeat_metricbeat"
      },
      "name": "metricbeat",
      "runtime": "docker"
     },
     "docker": {
      "container": {
       "command": "/usr/bin/tini -- /usr/local/bin/docker-entrypoint -environment container",
       "created": "2023-01-10T15:36:28.000Z",
       "ip_addresses": [
        "172.21.0.2"
       ],
       "labels": {
        "com_docker_compose_config-hash": "b3649497f15da387a04a0ba60abc6dda1b906e54b14cf1483e0026e36500a512",
        "com_docker_compose_container-number": "1",
        "com_docker_compose_oneoff": "False",
        "com_docker_compose_project": "metricbeat",
        "com_docker_compose_project_config_files": "docker-compose.yml",
        "com_docker_compose_project_working_dir": "/home/pi/metricbeat",
        "com_docker_compose_service": "metricbeat",
        "com_docker_compose_version": "1.25.0",
        "description": "Metricbeat is a lightweight shipper for metrics.",
        "io_k8s_description": "Metricbeat is a lightweight shipper for metrics.",
        "io_k8s_display-name": "Metricbeat image",
        "license": "Elastic License",
        "maintainer": "infra@elastic.co",
        "name": "metricbeat",
        "org_label-schema_build-date": "2022-12-04T04:51:03Z",
        "org_label-schema_license": "Elastic License",
        "org_label-schema_name": "metricbeat",
        "org_label-schema_schema-version": "1.0",
        "org_label-schema_url": "https://www.elastic.co/beats/metricbeat",
        "org_label-schema_vcs-ref": "6d03209df870c63ef9d59d609268c11dfdc835dd",
        "org_label-schema_vcs-url": "github.com/elastic/beats/v7",
        "org_label-schema_vendor": "Elastic",
        "org_label-schema_version": "8.5.3",
        "org_opencontainers_image_created": "2022-12-04T04:51:03Z",
        "org_opencontainers_image_licenses": "Elastic License",
        "org_opencontainers_image_title": "Metricbeat",
        "org_opencontainers_image_vendor": "Elastic",
        "release": "1",
        "summary": "metricbeat",
        "url": "https://www.elastic.co/beats/metricbeat",
        "vendor": "Elastic",
        "version": "8.5.3"
       },
       "size": {
        "root_fs": 0,
        "rw": 0
       },
       "status": "Up 59 minutes"
      }
     },
     "event": {
      "dataset": "docker.container",
      "duration": 13095583,
      "module": "docker"
     },
     "metricset": {
      "name": "container",
      "period": 10000
     },
     "service": {
      "address": "unix:///var/run/docker.sock",
      "type": "docker"
     }
    }

  cpu...OK
    result:
    {
     "@timestamp": "2023-01-10T16:36:07.123Z",
     "container": {
      "cpu": {
       "usage": 0.0013759124087591242
      },
      "id": "9d41cbfce6ea05fa2192c9fce8f1db33339390f763e58ab798cb43572ae9af9e",
      "image": {
       "name": "elastic-agent_elastic-agent"
      },
      "name": "elastic-agent",
      "runtime": "docker"
     },
     "docker": {
      "container": {
       "labels": {
        "com_docker_compose_config-hash": "63e4287ebcbe6ac2019cfaf831ff5bf9ab087163358b51d4712c9de468d3f174",
        "com_docker_compose_container-number": "1",
        "com_docker_compose_oneoff": "False",
        "com_docker_compose_project": "elastic-agent",
        "com_docker_compose_project_config_files": "docker-compose.yml",
        "com_docker_compose_project_working_dir": "/home/pi/elastic-agent",
        "com_docker_compose_service": "elastic-agent",
        "com_docker_compose_version": "1.25.0",
        "description": "Agent manages other beats based on configuration provided.",
        "io_k8s_description": "Agent manages other beats based on configuration provided.",
        "io_k8s_display-name": "Elastic-Agent image",
        "license": "Elastic License",
        "maintainer": "infra@elastic.co",
        "name": "elastic-agent",
        "org_label-schema_build-date": "2022-12-06T00:06:31Z",
        "org_label-schema_license": "Elastic License",
        "org_label-schema_name": "elastic-agent",
        "org_label-schema_schema-version": "1.0",
        "org_label-schema_url": "https://www.elastic.co/beats/elastic-agent",
        "org_label-schema_vcs-ref": "0e1a7396f356f8af5ad28c3c8e941256f50c86a2",
        "org_label-schema_vcs-url": "github.com/elastic/elastic-agent",
        "org_label-schema_vendor": "Elastic",
        "org_label-schema_version": "8.5.3",
        "org_opencontainers_image_created": "2022-12-06T00:06:31Z",
        "org_opencontainers_image_licenses": "Elastic License",
        "org_opencontainers_image_title": "Elastic-Agent",
        "org_opencontainers_image_vendor": "Elastic",
        "release": "1",
        "summary": "elastic-agent",
        "url": "https://www.elastic.co/beats/elastic-agent",
        "vendor": "Elastic",
        "version": "8.5.3"
       }
      },
      "cpu": {
       "core": {},
       "kernel": {
        "norm": {
         "pct": 0.0001411192214111922
        },
        "pct": 0.0005644768856447688,
        "ticks": 67300603000
       },
       "system": {
        "norm": {
         "pct": 1
        },
        "pct": 4,
        "ticks": 30896080000000
       },
       "total": {
        "norm": {
         "pct": 0.0013759124087591242
        },
        "pct": 0.005503649635036497
       },
       "user": {
        "norm": {
         "pct": 0.0012345498783454987
        },
        "pct": 0.004938199513381995,
        "ticks": 95724911000
       }
      }
     },
     "event": {
      "dataset": "docker.cpu",
      "duration": 2139403607,
      "module": "docker"
     },
     "metricset": {
      "name": "cpu",
      "period": 10000
     },
     "service": {
      "address": "unix:///var/run/docker.sock",
      "type": "docker"
     }
    }

  diskio...OK
    result:
    {
     "@timestamp": "2023-01-10T16:36:09.269Z",
     "container": {
      "disk": {
       "read": {
        "bytes": 176128
       },
       "write": {
        "bytes": 122880
       }
      },
      "id": "ecc5737d2190a5652557b61a6b41f0c7250ec3069d12b1dfa5830b798992e3aa",
      "image": {
       "name": "metricbeat_metricbeat"
      },
      "name": "metricbeat",
      "runtime": "docker"
     },
     "docker": {
      "container": {
       "labels": {
        "com_docker_compose_config-hash": "b3649497f15da387a04a0ba60abc6dda1b906e54b14cf1483e0026e36500a512",
        "com_docker_compose_container-number": "1",
        "com_docker_compose_oneoff": "False",
        "com_docker_compose_project": "metricbeat",
        "com_docker_compose_project_config_files": "docker-compose.yml",
        "com_docker_compose_project_working_dir": "/home/pi/metricbeat",
        "com_docker_compose_service": "metricbeat",
        "com_docker_compose_version": "1.25.0",
        "description": "Metricbeat is a lightweight shipper for metrics.",
        "io_k8s_description": "Metricbeat is a lightweight shipper for metrics.",
        "io_k8s_display-name": "Metricbeat image",
        "license": "Elastic License",
        "maintainer": "infra@elastic.co",
        "name": "metricbeat",
        "org_label-schema_build-date": "2022-12-04T04:51:03Z",
        "org_label-schema_license": "Elastic License",
        "org_label-schema_name": "metricbeat",
        "org_label-schema_schema-version": "1.0",
        "org_label-schema_url": "https://www.elastic.co/beats/metricbeat",
        "org_label-schema_vcs-ref": "6d03209df870c63ef9d59d609268c11dfdc835dd",
        "org_label-schema_vcs-url": "github.com/elastic/beats/v7",
        "org_label-schema_vendor": "Elastic",
        "org_label-schema_version": "8.5.3",
        "org_opencontainers_image_created": "2022-12-04T04:51:03Z",
        "org_opencontainers_image_licenses": "Elastic License",
        "org_opencontainers_image_title": "Metricbeat",
        "org_opencontainers_image_vendor": "Elastic",
        "release": "1",
        "summary": "metricbeat",
        "url": "https://www.elastic.co/beats/metricbeat",
        "vendor": "Elastic",
        "version": "8.5.3"
       }
      },
      "diskio": {
       "read": {
        "bytes": 176128,
        "ops": 0,
        "queued": 0,
        "rate": 0,
        "service_time": 0,
        "wait_time": 0
       },
       "summary": {
        "bytes": 0,
        "ops": 0,
        "queued": 0,
        "rate": 0,
        "service_time": 0,
        "wait_time": 0
       },
       "write": {
        "bytes": 122880,
        "ops": 0,
        "queued": 0,
        "rate": 0,
        "service_time": 0,
        "wait_time": 0
       }
      }
     },
     "event": {
      "dataset": "docker.diskio",
      "duration": 2133323204,
      "module": "docker"
     },
     "metricset": {
      "name": "diskio",
      "period": 10000
     },
     "service": {
      "address": "unix:///var/run/docker.sock",
      "type": "docker"
     }
    }

  event...
    error... ERROR timeout waiting for an event
  healthcheck...OK
    result:
    {
        ...
    }

  info...OK
    result:
    {
        ...
    }

  image...OK
    result:
    {
        ...
    }

  memory...OK
    result:
    {
        ...
    }

  network...OK
    result:
    {
        ...
    }

  network_summary...
    error... ERROR error fetching namespace for PID 1424: error reading network namespace link: readlink /proc/1424/ns/net: no such file or directory
system...
  cpu...OK
    result:
    {
        ...
    }

  load...OK
    result:
    {
        ...
    }

  memory...OK
    result:
    {
        ...
    }

  network...OK
    result:
    {
        ...
    }

  process...
    error... ERROR timeout waiting for an event
  process_summary...OK
    result:
    {
        ...
    }

  socket_summary...OK
    result:
    {
        ...
    }

system...
  filesystem...OK
    result:
    {
        ...
    }

  fsstat...OK
    result:
    {
        ...
    }

system...
  uptime...OK
    result:
    {
        ...
    }

I deleted most of the results but left all the errors and the diskio... results - as there are some metrics (diskio, summary, write) which are defenetly wrong.

Hope that helps somehow to identify the root cause?

11

When you run the metricbeat container, are you specifying this as argument to container:

spec:
  template:
    spec:
      containers:
          - args: [
            "-e",
            "-system.hostfs=/hostfs"
            ]
            "securityContext":
              "runAsUser": 0

Also you need to mount volume /proc and /sys/fs/cgroup to your container. See: Kubespray-4RPI4/09-metricbeat.yml at main · bpcvdhelm/Kubespray-4RPI4 (github.com) for metricbeat configuration on Raspberry PI.

12

Yes I mounted those filesystems within the container.

This is my docker-compose.yml:

version: '3.5'

secrets:
  ca.crt:
    file: ./cert/ca.crt

services:
  metricbeat:
    container_name: metricbeat
    hostname: openhab_metricbeat
    build:
      context: ./
    restart: unless-stopped
    user: root
    command:
      - -e
      - --strict.perms=false
    environment:
      ELASTICSEARCH_CA: '/ca.crt'
      ELASTICSEARCH_HOSTS: 'https://elasticsearch.my.domain:9200'
      ELASTICSEARCH_USERNAME: '****************'
      ELASTICSEARCH_PASSWORD: '****************'
      KIBANA_HOST: 'https://kibana.my.domain:5601'
    volumes:
      - ./config/metricbeat.docker.yml:/usr/share/metricbeat/metricbeat.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /proc:/hostfs/proc:ro
      - /:/hostfs:ro
    secrets:
      - source: ca.crt
        target: /ca.crt
    networks:
      - elk


networks:
  elk:
    driver: bridge

Passing the "-system.hostfs=/hostfs" as command just produces an errormessage telling me that this is not supported anymore and I supposed to use the config file. Hence I updated the config file which looks like this:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  path: ${system.hostfs}/hostfs
  reload.enabled: false

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

metricbeat.modules:

#-------------------------------- System Module --------------------------------
- module: system
  metricsets:
    - cpu             # CPU usage
    - load            # CPU load averages
    - memory          # Memory usage
    - network         # Network IO
    - process         # Per process metrics
    - process_summary # Process summary
    - uptime          # System Uptime
    - socket_summary  # Socket summary
    - core           # Per CPU core usage
    - diskio         # Disk IO
    - filesystem     # File system usage for each mountpoint
    - fsstat         # File system summary metrics
    #- raid           # Raid
    - socket         # Sockets and connection info (linux only)
    #- service        # systemd service information
  enabled: true
  period: 10s
  processes: ['.*']

- module: docker
  metricsets:
    - "container"
    - "cpu"
    - "diskio"
    - "event"
    - "healthcheck"
    - "info"
    - "image"
    - "memory"
    - "network"
    - "network_summary"
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true

# ================================== General ===================================
tags: ["openhab", "docker"]

With these updates I am still don't get the details from disk-io and in the logs I find this error messages:

{"log.level":"error","@timestamp":"2023-01-12T09:07:00.197Z","log.origin":{"file.name":"module/wrapper.go","file.line":263},"message":"Error fetching data for metricset docker.network_summary: error fetching namespace for PID 1206: error reading network namespace link: readlink /proc/1206/ns/net: no such file or directory","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-12T09:07:53.385Z","log.logger":"metrics","log.origin":{"file.name":"report/report.go","file.line":234},"message":"error getting cgroup stats for V2: error fetching stats for controller io: error fetching IO stats: error getting io.stats for path /hostfs/sys/fs/cgroup: error scanning file: /hostfs/sys/fs/cgroup/io.stat: input does not match format","service.name":"metricbeat","ecs.version":"1.6.0"}

There are also a lots of warnings:

{"log.level":"warn","@timestamp":"2023-01-12T09:10:10.423Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 10, 7, 61010011, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"614c6767-850e-4f8e-b208-b3adfaabe295\",\"id\":\"d7c7dfa6-025a-4d96-b8f6-ae5fa6362aec\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"container\":{\"id\":\"cf2425a968152ad0d8a119da720f777fb69b70bc50574f8ac139ce483fe830e7\",\"image\":{\"name\":\"ghcr.io/jens-maus/raspberrymatic:latest\"},\"memory\":{\"usage\":0.13816561590189386},\"name\":\"ccu\",\"runtime\":\"docker\"},\"docker\":{\"container\":{\"labels\":{\"com_datadoghq_ad_logs\":\"[{\\\"source\\\": \\\"raspberrymatic\\\", \\\"service\\\": \\\"raspberrymatic\\\"}]\",\"com_docker_compose_config-hash\":\"a987e3a8d753f42ddb1cdb844ced8e93d734c4f4b64d111b2b67e0012ab46779\",\"com_docker_compose_container-number\":\"1\",\"com_docker_compose_oneoff\":\"False\",\"com_docker_compose_project\":\"docker\",\"com_docker_compose_project_config_files\":\"docker-compose.yaml\",\"com_docker_compose_project_working_dir\":\"/home/pi/docker\",\"com_docker_compose_service\":\"raspberrymatic\",\"com_docker_compose_version\":\"1.25.0\",\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"}},\"memory\":{\"fail\":{\"count\":0},\"limit\":3978674176,\"rss\":{\"pct\":0,\"total\":0},\"stats\":{\"active_anon\":9695232,\"active_file\":105422848,\"anon\":409272320,\"anon_thp\":0,\"file\":153096192,\"file_dirty\":0,\"file_mapped\":25309184,\"file_writeback\":0,\"inactive_anon\":409288704,\"inactive_file\":37974016,\"kernel_stack\":2015232,\"pgactivate\":38042,\"pgdeactivate\":10736,\"pgfault\":23461057,\"pglazyfree\":0,\"pglazyfreed\":0,\"pgmajfault\":530,\"pgrefill\":15905,\"pgscan\":117876,\"pgsteal\":33783,\"shmem\":9699328,\"slab\":21011752,\"slab_reclaimable\":18228888,\"slab_unreclaimable\":2782864,\"sock\":36864,\"thp_collapse_alloc\":0,\"thp_fault_alloc\":0,\"unevictable\":0,\"workingset_activate\":0,\"workingset_nodereclaim\":0,\"workingset_refault\":0},\"usage\":{\"max\":0,\"pct\":0.13816561590189386,\"total\":549715968}}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.memory\",\"duration\":2109356618,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"memory\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.container.labels.org_opencontainers_image_created] of type [date] in document with id 'WXI-pYUBBZFRz5sZbkmE'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-01-12T09:10:10.423Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 10, 9, 837451224, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"614c6767-850e-4f8e-b208-b3adfaabe295\",\"id\":\"d7c7dfa6-025a-4d96-b8f6-ae5fa6362aec\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"docker\":{\"image\":{\"created\":\"2022-12-18T09:09:28.000Z\",\"id\":{\"current\":\"sha256:3b270c8f016295f86115f5e6c340deb4f225a5270858935220364c7889c4edfb\",\"parent\":\"\"},\"labels\":{\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"},\"size\":{\"regular\":488336932,\"virtual\":488336932},\"tags\":[\"ghcr.io/jens-maus/raspberrymatic:latest\"]}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.image\",\"duration\":82741571,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"image\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.image.labels.org_opencontainers_image_created] of type [date] in document with id 'YnI-pYUBBZFRz5sZbkmE'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-01-12T09:10:12.915Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 10, 9, 249664408, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"614c6767-850e-4f8e-b208-b3adfaabe295\",\"id\":\"d7c7dfa6-025a-4d96-b8f6-ae5fa6362aec\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"container\":{\"disk\":{\"read\":{\"bytes\":236494848},\"write\":{\"bytes\":64557056}},\"id\":\"cf2425a968152ad0d8a119da720f777fb69b70bc50574f8ac139ce483fe830e7\",\"image\":{\"name\":\"ghcr.io/jens-maus/raspberrymatic:latest\"},\"name\":\"ccu\",\"runtime\":\"docker\"},\"docker\":{\"container\":{\"labels\":{\"com_datadoghq_ad_logs\":\"[{\\\"source\\\": \\\"raspberrymatic\\\", \\\"service\\\": \\\"raspberrymatic\\\"}]\",\"com_docker_compose_config-hash\":\"a987e3a8d753f42ddb1cdb844ced8e93d734c4f4b64d111b2b67e0012ab46779\",\"com_docker_compose_container-number\":\"1\",\"com_docker_compose_oneoff\":\"False\",\"com_docker_compose_project\":\"docker\",\"com_docker_compose_project_config_files\":\"docker-compose.yaml\",\"com_docker_compose_project_working_dir\":\"/home/pi/docker\",\"com_docker_compose_service\":\"raspberrymatic\",\"com_docker_compose_version\":\"1.25.0\",\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"}},\"diskio\":{\"read\":{\"bytes\":236494848,\"ops\":0,\"queued\":0,\"rate\":0,\"service_time\":0,\"wait_time\":0},\"summary\":{\"bytes\":0,\"ops\":0,\"queued\":0,\"rate\":0,\"service_time\":0,\"wait_time\":0},\"write\":{\"bytes\":64557056,\"ops\":0,\"queued\":0,\"rate\":0,\"service_time\":0,\"wait_time\":0}}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.diskio\",\"duration\":2014061721,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"diskio\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.container.labels.org_opencontainers_image_created] of type [date] in document with id 'fnI-pYUBBZFRz5sZdkqu'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-01-12T09:10:14.907Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 10, 11, 349918713, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"614c6767-850e-4f8e-b208-b3adfaabe295\",\"id\":\"d7c7dfa6-025a-4d96-b8f6-ae5fa6362aec\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"container\":{\"id\":\"cf2425a968152ad0d8a119da720f777fb69b70bc50574f8ac139ce483fe830e7\",\"image\":{\"name\":\"ghcr.io/jens-maus/raspberrymatic:latest\"},\"name\":\"ccu\",\"network\":{\"egress\":{\"bytes\":80924287},\"ingress\":{\"bytes\":34736784}},\"runtime\":\"docker\"},\"docker\":{\"container\":{\"labels\":{\"com_datadoghq_ad_logs\":\"[{\\\"source\\\": \\\"raspberrymatic\\\", \\\"service\\\": \\\"raspberrymatic\\\"}]\",\"com_docker_compose_config-hash\":\"a987e3a8d753f42ddb1cdb844ced8e93d734c4f4b64d111b2b67e0012ab46779\",\"com_docker_compose_container-number\":\"1\",\"com_docker_compose_oneoff\":\"False\",\"com_docker_compose_project\":\"docker\",\"com_docker_compose_project_config_files\":\"docker-compose.yaml\",\"com_docker_compose_project_working_dir\":\"/home/pi/docker\",\"com_docker_compose_service\":\"raspberrymatic\",\"com_docker_compose_version\":\"1.25.0\",\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"}},\"network\":{\"inbound\":{\"bytes\":34736784,\"dropped\":0,\"errors\":0,\"packets\":171201},\"interface\":\"eth0\",\"outbound\":{\"bytes\":80924287,\"dropped\":0,\"errors\":0,\"packets\":266506}}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.network\",\"duration\":2014493548,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"network\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.container.labels.org_opencontainers_image_created] of type [date] in document with id 'MnI-pYUBBZFRz5sZf0sf'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-01-12T09:10:17.905Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 10, 16, 533410422, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"614c6767-850e-4f8e-b208-b3adfaabe295\",\"id\":\"d7c7dfa6-025a-4d96-b8f6-ae5fa6362aec\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"container\":{\"id\":\"cf2425a968152ad0d8a119da720f777fb69b70bc50574f8ac139ce483fe830e7\",\"image\":{\"name\":\"ghcr.io/jens-maus/raspberrymatic:latest\"},\"name\":\"ccu\",\"runtime\":\"docker\"},\"docker\":{\"container\":{\"command\":\"/sbin/init\",\"created\":\"2023-01-02T10:00:25.000Z\",\"ip_addresses\":[\"172.18.0.2\"],\"labels\":{\"com_datadoghq_ad_logs\":\"[{\\\"source\\\": \\\"raspberrymatic\\\", \\\"service\\\": \\\"raspberrymatic\\\"}]\",\"com_docker_compose_config-hash\":\"a987e3a8d753f42ddb1cdb844ced8e93d734c4f4b64d111b2b67e0012ab46779\",\"com_docker_compose_container-number\":\"1\",\"com_docker_compose_oneoff\":\"False\",\"com_docker_compose_project\":\"docker\",\"com_docker_compose_project_config_files\":\"docker-compose.yaml\",\"com_docker_compose_project_working_dir\":\"/home/pi/docker\",\"com_docker_compose_service\":\"raspberrymatic\",\"com_docker_compose_version\":\"1.25.0\",\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"},\"size\":{\"root_fs\":0,\"rw\":0},\"status\":\"Up 43 hours (healthy)\"}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.container\",\"duration\":11304304,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"container\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.container.labels.org_opencontainers_image_created] of type [date] in document with id 'QXI-pYUBBZFRz5sZikxx'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"}
{

it look like that those mainly related to the docker module.

13

I thought it might be helpful to see the startup log of the container and therefor deleted the container and captured the logs right from the beginning:

{"log.level":"info","@timestamp":"2023-01-12T09:20:36.306Z","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:36.329Z","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: a30ffb34-5553-404e-ab63-c61016f74379","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-01-12T09:20:39.331Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.374Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.375Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1082},"message":"Beat info","service.name":"metricbeat","system_info":{"beat":{"path":{"config":"/usr/share/metricbeat","data":"/usr/share/metricbeat/data","home":"/usr/share/metricbeat","logs":"/usr/share/metricbeat/logs"},"type":"metricbeat","uuid":"a30ffb34-5553-404e-ab63-c61016f74379"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.375Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1091},"message":"Build info","service.name":"metricbeat","system_info":{"build":{"commit":"6d03209df870c63ef9d59d609268c11dfdc835dd","libbeat":"8.5.3","time":"2022-12-04T04:51:35.000Z","version":"8.5.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.375Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1094},"message":"Go runtime info","service.name":"metricbeat","system_info":{"go":{"os":"linux","arch":"arm64","max_procs":4,"version":"go1.18.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.378Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1098},"message":"Host info","service.name":"metricbeat","system_info":{"host":{"architecture":"aarch64","boot_time":"2023-01-10T14:26:35Z","containerized":false,"name":"openhab_metricbeat","ip":["127.0.0.1/8","172.21.0.2/16"],"kernel_version":"5.15.84-v8+","mac":["02:42:ac:15:00:02"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.5 LTS (Focal Fossa)","major":20,"minor":4,"patch":5,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.379Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1127},"message":"Process info","service.name":"metricbeat","system_info":{"process":{"capabilities":{"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/metricbeat","exe":"/usr/share/metricbeat/metricbeat","name":"metricbeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2023-01-12T09:20:35.400Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.380Z","log.origin":{"file.name":"instance/beat.go","file.line":294},"message":"Setup Beat: metricbeat; Version: 8.5.3","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.387Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://elasticsearch.my.domain:9200","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.388Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: openhab_metricbeat","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.411Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":102},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.476Z","log.origin":{"file.name":"filesystem/filesystem.go","file.line":70},"message":"Ignoring filesystem types: sysfs, tmpfs, bdev, proc, cgroup, cgroup2, cpuset, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, bpf, pipefs, ramfs, rpc_pipefs, devpts, nfs, nfs4, autofs, mqueue, binder, pstore, fuse, fusectl, overlay","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.477Z","log.logger":"system.fsstat","log.origin":{"file.name":"fsstat/fsstat.go","file.line":60},"message":"Ignoring filesystem types: %ssysfs, tmpfs, bdev, proc, cgroup, cgroup2, cpuset, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, bpf, pipefs, ramfs, rpc_pipefs, devpts, nfs, nfs4, autofs, mqueue, binder, pstore, fuse, fusectl, overlay","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.480Z","log.origin":{"file.name":"socket/socket.go","file.line":83},"message":"socket process info will only be available for processes owned by the 0 user because this Beat is not running with enough privileges","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-01-12T09:20:39.484Z","log.logger":"cfgwarn","log.origin":{"file.name":"network_summary/network_summary.go","file.line":58},"message":"BETA: The docker network_summary metricset is beta.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.486Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Starting metrics logging every 30s","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.486Z","log.origin":{"file.name":"instance/beat.go","file.line":471},"message":"metricbeat start running.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.487Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"Config reloader started","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:39.488Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:40.581Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(https://elasticsearch.my.domain:9200))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:40.661Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.5.3","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:40.674Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.5.3","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:40.674Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":231},"message":"Auto ILM enable success.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:40.685Z","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy metricbeat exists already.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:40.686Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":366},"message":"Set settings.index.lifecycle.name in template to {metricbeat {\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_age\":\"30d\",\"max_primary_shard_size\":\"50gb\"}}}}}}} as ILM is enabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:40.701Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":115},"message":"Template \"metricbeat-8.5.3\" already exists and will not be overwritten.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:40.701Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":267},"message":"Loaded index template.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-12T09:20:40.704Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(https://elasticsearch.my.domain:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-12T09:20:46.019Z","log.origin":{"file.name":"module/wrapper.go","file.line":263},"message":"Error fetching data for metricset docker.network_summary: error fetching namespace for PID 7956: error reading network namespace link: readlink /proc/7956/ns/net: no such file or directory","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-01-12T09:20:52.504Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 12, 9, 20, 40, 835834909, time.Local), Meta:null, Fields:{\"agent\":{\"ephemeral_id\":\"de227a3d-f661-4565-8f3d-8c1392bebc8e\",\"id\":\"a30ffb34-5553-404e-ab63-c61016f74379\",\"name\":\"openhab_metricbeat\",\"type\":\"metricbeat\",\"version\":\"8.5.3\"},\"container\":{\"id\":\"cf2425a968152ad0d8a119da720f777fb69b70bc50574f8ac139ce483fe830e7\",\"image\":{\"name\":\"ghcr.io/jens-maus/raspberrymatic:latest\"},\"name\":\"ccu\",\"runtime\":\"docker\"},\"docker\":{\"container\":{\"command\":\"/sbin/init\",\"created\":\"2023-01-02T10:00:25.000Z\",\"ip_addresses\":[\"172.18.0.2\"],\"labels\":{\"com_datadoghq_ad_logs\":\"[{\\\"source\\\": \\\"raspberrymatic\\\", \\\"service\\\": \\\"raspberrymatic\\\"}]\",\"com_docker_compose_config-hash\":\"a987e3a8d753f42ddb1cdb844ced8e93d734c4f4b64d111b2b67e0012ab46779\",\"com_docker_compose_container-number\":\"1\",\"com_docker_compose_oneoff\":\"False\",\"com_docker_compose_project\":\"docker\",\"com_docker_compose_project_config_files\":\"docker-compose.yaml\",\"com_docker_compose_project_working_dir\":\"/home/pi/docker\",\"com_docker_compose_service\":\"raspberrymatic\",\"com_docker_compose_version\":\"1.25.0\",\"io_hass_arch\":\"armhf|aarch64|amd64\",\"io_hass_description\":\"HomeMatic/homematicIP CCU central based on RaspberryMatic\",\"io_hass_name\":\"RaspberryMatic CCU\",\"io_hass_type\":\"addon\",\"io_hass_url\":\"https://github.com/jens-maus/RaspberryMatic/tree/master/home-assistant-addon\",\"io_hass_version\":\"3.65.11.20221218\",\"org_opencontainers_image_authors\":\"RaspberryMatic OpenSource Team\",\"org_opencontainers_image_created\":\"2022-12-18 06:23:30\",\"org_opencontainers_image_description\":\"Alternative OS for your HomeMatic CCU\",\"org_opencontainers_image_documentation\":\"https://github.com/jens-maus/RaspberryMatic/wiki\",\"org_opencontainers_image_licenses\":\"Apache-2.0\",\"org_opencontainers_image_ref_name\":\"master\",\"org_opencontainers_image_revision\":\"75d664c83dffe149dfcf26ba995c368438648b88\",\"org_opencontainers_image_source\":\"https://github.com/jens-maus/RaspberryMatic\",\"org_opencontainers_image_title\":\"RaspberryMatic\",\"org_opencontainers_image_url\":\"https://raspberrymatic.de\",\"org_opencontainers_image_vendor\":\"RasperryMatic OpenSource Project\",\"org_opencontainers_image_version\":\"3.65.11.20221218\"},\"size\":{\"root_fs\":0,\"rw\":0},\"status\":\"Up 43 hours (healthy)\"}},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"dataset\":\"docker.container\",\"duration\":13389279,\"module\":\"docker\"},\"host\":{\"name\":\"openhab_metricbeat\"},\"metricset\":{\"name\":\"container\",\"period\":10000},\"service\":{\"address\":\"unix:///var/run/docker.sock\",\"type\":\"docker\"},\"tags\":[\"openhab\",\"docker\"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [docker.container.labels.org_opencontainers_image_created] of type [date] in document with id 'p3JIpYUBBZFRz5sZOb4l'. Preview of field's value: '2022-12-18 06:23:30'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2022-12-18 06:23:30] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Failed to parse with all enclosed parsers\"}}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"}
14

Since this format is not supported implicitly by ES, guess you need to define your own format in your index template to avoid ingestion failure issues: format | Elasticsearch Guide [8.6] | Elastic
About the other issue where metrics are not being polled, guess there was another issue opened without any responses: Metricbeat error about scanning empty io.stat file - Elastic Stack / Beats - Discuss the Elastic Stack

And I see you have already commented on github issue for this :slight_smile:

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.