Failed to load BPF probes

Elastic Security
1

Hello, does anyone know how to resolve this issue?

My kernel version is 6.4.5-1. el7.elrepo.x86_ 64, the k8s version is v1.23.6, the system is CentOS Linux 7 (Core), and the Docker version is 24.0.5

state: 3
message: 1 or more components/units in a failed state
fleet_state: 2
fleet_message: Connected
log_level: info
components:
- id: cloud_defend/control-default
  state:
    state: 2
    message: 'Healthy: communicating with pid ''7301'''
    units:
      ? unittype: 0
        unitid: cloud_defend/control-default-cloud_defend/control-cloud_defend-43d2f730-2b88-11ee-8b45-a5b2911a0150
      : state: 4
        message: 'config update: 1 failures'
        payload:
          errors:
          - component: bpf-sensor
            message: Failed to load BPF probes
            payload:
              log: failed BPF state
      ? unittype: 1
        unitid: cloud_defend/control-default
      : state: 4
        message: 'config update: 1 failures'
        payload:
          errors:
          - component: bpf-sensor
            message: Failed to load BPF probes
            payload:
              log: failed BPF state
    features_idx: 1
    version_info:
      name: cloud-defend
      version: 8.8.2-076abf2
2

It seems to be incompatible with my kernel

{"log.level":"error","@timestamp":"2023-07-31T08:58:55.416Z","message":"load bpf progs (Please ensure required Linux capabilities are set in k8s security context. BPF, PERFMON, and SYS_RESOURCE are mandatory): error loading eBPF probes: field KprobeTaskstatsExit: program kprobe__taskstats_exit: apply CO-RE relocations: load kernel spec: no BTF found for kernel version 6.4.1-1.el7.elrepo.x86_64: not supported","component":{"binary":"cloud-defend","dataset":"elastic_agent.cloud_defend","id":"cloud_defend/control-default","type":"cloud_defend/control"},"log":{"source":"cloud_defend/control-default"},"log.logger":"ebpf-sensor","ecs.version":"1.6.0","ecs.version":"1.6.0"}
3

Hello @L1NG,

We appreciate your interest in Defend for Containers and are excited to help you as you deploy the software.

Currently, we are testing/supporting the integration solely in AWS and GCP Kubernetes environments. This means most of our testing is completed using Container optimized OS or Amazon Linux.

We would love to support your use case; however, the BPF probes require a Linux kernel with BTF enabled. BTF is a BPF feature that allows generic BPF programs to run across multiple kernel compilations.

The elrepo kernel distribution does not have BTF enabled. Is it possible for you to test on a distribution that supports BTF? This requires the kernel to be compiled with CONFIG_DEBUG_INFO_BTF enabled. Most major distributions have kernels that do support this out of the box.

Some additional requirements of the Defend for Containers integration are that it is deployed on a 5.10.50 (or newer) kernel and is running Kubernetes 1.24 or newer.

4

Hello, I downloaded Linux 5.15.63-1. el7.x86_ 64 from the Linux community, but it doesn't seem to support BTF either. Do I need to download the kernel source code and recompile it myself? Or can you provide me with a kernel version that supports BTF out of the box.
image

5

Hi L1NG,

For CentOS Linux 7, I am unaware of a public kernel that would work or you out of the box.

Compiling a kernel with the necessary configuration is also an option; however, I cannot guarantee your success as it is not one of our tested environments.

I would recommend testing in one of the environments mentioned above before investing much time in recompiling a kernel.

6

Hi @Norrie,
I have successfully compiled kernel 5.15.127 and enabled CONFIG_ DEBUG_ INFO_ BTF=y, but the error shows that I seem to need other kernel modules or capabilities. What else do I need to enable?

image
image760×69 3.54 KB

Here is the error message

{"log.level":"error","@timestamp":"2023-08-24T00:50:15.019Z","message":"load bpf progs (Please ensure required Linux capabilities are set in k8s security context. BPF, PERFMON, and SYS_RESOURCE are mandatory): error loading eBPF probes: field LsmFileOpen: program lsm__file_open: attach LSM/LSMMac: find target for file_open LSM hook in modules: iterate modules: get next BTF ID: operation not permitted","component":{"binary":"cloud-defend","dataset":"elastic_agent.cloud_defend","id":"cloud_defend/control-default","type":"cloud_defend/control"},"log":{"source":"cloud_defend/control-default"},"log.logger":"ebpf-sensor","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Can you provide the kernel's config file

7

Hello @Norrie , I am trying to install and deploy in the environment you mentioned. How can I verify if the plugin is effective？I am currently displaying the agent integration plugin as normal

8

First, you will want to check if the integration produces logs. You can observe this by navigating to the "Discover" view in Kibana and selecting the "logs-cloud_defend*" data view. If you see logs, then you know the integration is producing events.

Screenshot 2023-08-31 at 1.54.55 PM
Screenshot 2023-08-31 at 1.54.55 PM1920×861 192 KB

Once you have confirmed this, you can navigate to the "Kubernetes dashboard" in the security solution for better data visualization.

Screenshot 2023-08-31 at 1.59.49 PM
Screenshot 2023-08-31 at 1.59.49 PM3794×1714 502 KB

9

Hello, did you test using EKS Kubernetes Worker AMI with Amazon Linux2 image? I am using this image but will report an error as follows

image
image1238×485 48.4 KB

environment：ami-08fd36634c00dfab3
amazon/amazon-eks-node-1.22-v20220429
EKS Kubernetes Worker AMI with AmazonLinux2 image, (k8s: 1.22.6, docker: 20.10.13-2.amzn2, containerd: 1.4.13-2.amzn2.0.1)

10

Hello @L1NG.

It looks like you are making progress. We do testing EKS; however, as mentioned above, the minimum version we support is 1.24. 1.22 is no longer supported by AWS, so I recommend upgrading your cluster.

Please double-check that you are uncommenting the following lines in the YAML defining your K8s agent manifest.

          securityContext:
            runAsUser: 0
            # The following capabilities are needed for 'Defend for containers' integration (cloud-defend)
            # If you are using this integration, please uncomment these lines before applying.
            capabilities:
              add:
                - BPF # (since Linux 5.8) allows loading of BPF programs, create most map types, load BTF, iterate programs and maps.
                - PERFMON # (since Linux 5.8) allows attaching of BPF programs used for performance metrics and observability operations.
                - SYS_RESOURCE # Allow use of special resources or raising of resource limits. Used by 'Defend for Containers' to modify 'rlimit_memlock'
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.