Packetbeat 8.5 high CPU usage after updade from 7.17.7

[Alexander_A]

After packetbeat update from 7.17.7 to 8.5.0 all our hosts show high CPU usage.
Packetbeat used only for DNS traffic screening. Even after we enabled only ICMP in config we see more than 15% CPU usage but before it was less than 2%.

After downgrade to version 7.17.7 everything works as expected.

Packetbeat config:

packetbeat.interfaces.device: ens160

packetbeat.interfaces.internal_networks:
  - private

packetbeat.flows:
  enabled: false

packetbeat.protocols:
- type: icmp
  enabled: true

processors:
  - # Add forwarded to tags when processing data from a network tap or mirror.
    if.contains.tags: forwarded
    then:
      - drop_fields:
          fields: [host]
    else:
      - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - detect_mime_type:
      field: http.request.body.content
      target: http.request.mime_type
  - detect_mime_type:
      field: http.response.body.content
      target: http.response.mime_type

logging.to_files: true
logging.metrics.enabled: true

output.kafka:
  hosts: ["kafka-elk:9092"]
  topic: "packetbeat"
  required_acks: 1
  compression: gzip

Packetbeat metrics:

{
  "log.level": "info",
  "@timestamp": "2022-11-29T17:23:33.446+0300",
  "log.logger": "monitoring",
  "log.origin": {
    "file.name": "log/log.go",
    "file.line": 186
  },
  "message": "Non-zero metrics in the last 30s",
  "service.name": "packetbeat",
  "monitoring": {
    "metrics": {
      "beat": {
        "cgroup": {
          "memory": {
            "mem": {
              "usage": {
                "bytes": 54718464
              }
            }
          }
        },
        "cpu": {
          "system": {
            "ticks": 44470,
            "time": {
              "ms": 790
            }
          },
          "total": {
            "ticks": 323480,
            "time": {
              "ms": 5400
            },
            "value": 323480
          },
          "user": {
            "ticks": 279010,
            "time": {
              "ms": 4610
            }
          }
        },
        "handles": {
          "limit": {
            "hard": 524288,
            "soft": 1024
          },
          "open": 12
        },
        "info": {
          "ephemeral_id": "1970db0e-32f4-48f3-81f6-69cff4074dec",
          "uptime": {
            "ms": 1893050
          },
          "version": "8.5.0"
        },
        "memstats": {
          "gc_next": 16463480,
          "memory_alloc": 12145720,
          "memory_total": 68349915128,
          "rss": 121389056
        },
        "runtime": {
          "goroutines": 38
        }
      },
      "libbeat": {
        "config": {
          "module": {
            "running": 0
          }
        },
        "output": {
          "events": {
            "acked": 3,
            "active": 0,
            "batches": 2,
            "total": 3
          }
        },
        "outputs": {
          "kafka": {
            "bytes_read": 162,
            "bytes_write": 2582
          }
        },
        "pipeline": {
          "clients": 1,
          "events": {
            "active": 0,
            "published": 3,
            "total": 3
          },
          "queue": {
            "acked": 3
          }
        }
      },
      "system": {
        "load": {
          "1": 6.98,
          "15": 10.13,
          "5": 10.88,
          "norm": {
            "1": 0.5817,
            "15": 0.8442,
            "5": 0.9067
          }
        }
      }
    },
    "ecs.version": "1.6.0"
  }
}

memory_total value x600 bigger than for version of 7.17.7

In debug mode a lot of logs generated looking like:

{"log.level":"debug","@timestamp":"2022-11-29T08:38:50.626+0300","log.logger":"sniffer","log.origin":{"file.name":"sniffer/sniffer.go","file.line":437},"message":"Packet number: 806529","service.name":"packetbeat","ecs.version":"1.6.0"}

{"log.level":"debug","@timestamp":"2022-11-29T08:38:50.626+0300","log.logger":"decoder","log.origin":{"file.name":"decoder/decoder.go","file.line":174},"message":"decode packet data","service.name":"packetbeat","ecs.version":"1.6.0"}

{"log.level":"debug","@timestamp":"2022-11-29T08:38:50.626+0300","log.logger":"decoder","log.origin":{"file.name":"decoder/decoder.go","file.line":247},"message":"IPv4 packet","service.name":"packetbeat","ecs.version":"1.6.0"}

{"log.level":"debug","@timestamp":"2022-11-29T08:38:50.626+0300","log.logger":"decoder","log.origin":{"file.name":"decoder/decoder.go","file.line":288},"message":"TCP packet","service.name":"packetbeat","ecs.version":"1.6.0"}

{"log.level":"debug","@timestamp":"2022-11-29T08:38:50.626+0300","log.logger":"decoder","log.origin":{"file.name":"decoder/decoder.go","file.line":359},"message":"Ignore empty non-FIN packet","service.name":"packetbeat","ecs.version":"1.6.0"}

{"log.level":"debug","@timestamp":"2022-11-29T08:38:50.626+0300","log.logger":"sniffer","log.origin":{"file.name":"sniffer/sniffer.go","file.line":437},"message":"Packet number: 806530","service.name":"packetbeat","ecs.version":"1.6.0"} 

Link to the debug.log file - share/packetbeat-debug.log at main · leweafan/share · GitHub

Also after upgrade service became unresponsive and can not be stopped.

Nov 29 17:36:22 testhost systemd[1]: packetbeat.service: Failed with result 'timeout'.

That's how CPU usage changed after update:

OS: Debian 10.13

P.S. Upgrade to the latest version 8.5.2 doesn't help

[Alexander_A]

Tried different interface type with the same result

packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100