Failed to load BPF probes

L1NG · July 31, 2023, 9:13am

Hello, does anyone know how to resolve this issue?

My kernel version is 6.4.5-1. el7.elrepo.x86_ 64, the k8s version is v1.23.6, the system is CentOS Linux 7 (Core), and the Docker version is 24.0.5

state: 3
message: 1 or more components/units in a failed state
fleet_state: 2
fleet_message: Connected
log_level: info
components:
- id: cloud_defend/control-default
  state:
    state: 2
    message: 'Healthy: communicating with pid ''7301'''
    units:
      ? unittype: 0
        unitid: cloud_defend/control-default-cloud_defend/control-cloud_defend-43d2f730-2b88-11ee-8b45-a5b2911a0150
      : state: 4
        message: 'config update: 1 failures'
        payload:
          errors:
          - component: bpf-sensor
            message: Failed to load BPF probes
            payload:
              log: failed BPF state
      ? unittype: 1
        unitid: cloud_defend/control-default
      : state: 4
        message: 'config update: 1 failures'
        payload:
          errors:
          - component: bpf-sensor
            message: Failed to load BPF probes
            payload:
              log: failed BPF state
    features_idx: 1
    version_info:
      name: cloud-defend
      version: 8.8.2-076abf2

L1NG · July 31, 2023, 9:34am

It seems to be incompatible with my kernel

{"log.level":"error","@timestamp":"2023-07-31T08:58:55.416Z","message":"load bpf progs (Please ensure required Linux capabilities are set in k8s security context. BPF, PERFMON, and SYS_RESOURCE are mandatory): error loading eBPF probes: field KprobeTaskstatsExit: program kprobe__taskstats_exit: apply CO-RE relocations: load kernel spec: no BTF found for kernel version 6.4.1-1.el7.elrepo.x86_64: not supported","component":{"binary":"cloud-defend","dataset":"elastic_agent.cloud_defend","id":"cloud_defend/control-default","type":"cloud_defend/control"},"log":{"source":"cloud_defend/control-default"},"log.logger":"ebpf-sensor","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Norrie · July 31, 2023, 8:23pm

Hello @L1NG,

We appreciate your interest in Defend for Containers and are excited to help you as you deploy the software.

Currently, we are testing/supporting the integration solely in AWS and GCP Kubernetes environments. This means most of our testing is completed using Container optimized OS or Amazon Linux.

We would love to support your use case; however, the BPF probes require a Linux kernel with BTF enabled. BTF is a BPF feature that allows generic BPF programs to run across multiple kernel compilations.

The elrepo kernel distribution does not have BTF enabled. Is it possible for you to test on a distribution that supports BTF? This requires the kernel to be compiled with CONFIG_DEBUG_INFO_BTF enabled. Most major distributions have kernels that do support this out of the box.

Some additional requirements of the Defend for Containers integration are that it is deployed on a 5.10.50 (or newer) kernel and is running Kubernetes 1.24 or newer.

L1NG · August 7, 2023, 2:28am

Hello, I downloaded Linux 5.15.63-1. el7.x86_ 64 from the Linux community, but it doesn't seem to support BTF either. Do I need to download the kernel source code and recompile it myself? Or can you provide me with a kernel version that supports BTF out of the box.

Norrie · August 21, 2023, 6:39pm

Hi L1NG,

For CentOS Linux 7, I am unaware of a public kernel that would work or you out of the box.

Compiling a kernel with the necessary configuration is also an option; however, I cannot guarantee your success as it is not one of our tested environments.

I would recommend testing in one of the environments mentioned above before investing much time in recompiling a kernel.

L1NG · August 24, 2023, 1:10am

Hi @Norrie,
I have successfully compiled kernel 5.15.127 and enabled CONFIG_ DEBUG_ INFO_ BTF=y, but the error shows that I seem to need other kernel modules or capabilities. What else do I need to enable?

Here is the error message

{"log.level":"error","@timestamp":"2023-08-24T00:50:15.019Z","message":"load bpf progs (Please ensure required Linux capabilities are set in k8s security context. BPF, PERFMON, and SYS_RESOURCE are mandatory): error loading eBPF probes: field LsmFileOpen: program lsm__file_open: attach LSM/LSMMac: find target for file_open LSM hook in modules: iterate modules: get next BTF ID: operation not permitted","component":{"binary":"cloud-defend","dataset":"elastic_agent.cloud_defend","id":"cloud_defend/control-default","type":"cloud_defend/control"},"log":{"source":"cloud_defend/control-default"},"log.logger":"ebpf-sensor","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Can you provide the kernel's config file

L1NG · August 31, 2023, 3:29am

Hello @Norrie , I am trying to install and deploy in the environment you mentioned. How can I verify if the plugin is effective？I am currently displaying the agent integration plugin as normal

Norrie · August 31, 2023, 9:00pm

First, you will want to check if the integration produces logs. You can observe this by navigating to the "Discover" view in Kibana and selecting the "logs-cloud_defend*" data view. If you see logs, then you know the integration is producing events.

Once you have confirmed this, you can navigate to the "Kubernetes dashboard" in the security solution for better data visualization.

L1NG · September 5, 2023, 6:37am

Hello, did you test using EKS Kubernetes Worker AMI with Amazon Linux2 image? I am using this image but will report an error as follows

environment：ami-08fd36634c00dfab3
amazon/amazon-eks-node-1.22-v20220429
EKS Kubernetes Worker AMI with AmazonLinux2 image, (k8s: 1.22.6, docker: 20.10.13-2.amzn2, containerd: 1.4.13-2.amzn2.0.1)

Norrie · September 6, 2023, 6:28pm

Hello @L1NG.

It looks like you are making progress. We do testing EKS; however, as mentioned above, the minimum version we support is 1.24. 1.22 is no longer supported by AWS, so I recommend upgrading your cluster.

Please double-check that you are uncommenting the following lines in the YAML defining your K8s agent manifest.

          securityContext:
            runAsUser: 0
            # The following capabilities are needed for 'Defend for containers' integration (cloud-defend)
            # If you are using this integration, please uncomment these lines before applying.
            capabilities:
              add:
                - BPF # (since Linux 5.8) allows loading of BPF programs, create most map types, load BTF, iterate programs and maps.
                - PERFMON # (since Linux 5.8) allows attaching of BPF programs used for performance metrics and observability operations.
                - SYS_RESOURCE # Allow use of special resources or raising of resource limits. Used by 'Defend for Containers' to modify 'rlimit_memlock'

system · October 4, 2023, 6:28pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.