Packetbeat causing 100% CPU usage

[radu.capverde]

Hello,

We have multiple VMs that are running packetbeat and on some of them (8 VMs) the CPU usage is constantly at 100% due to packetbeat. We have had one instance where one VM became completely unresponsive due to this and were forced to reboot it to regain control.

After reboot the CPU stays within normal boundaries for a few hours, then again spikes to 100%.

All machines (those affected and those that are ok) are running CentOS 7.5 and packetbeat 5.6.12 with the same configuration.
Machines have no particularities or commonalities that may point to a common cause.

Please advise what logs or traces you need in order to be able to assist with this.

Regards,
Radu

[andrewkroh]

Is there anything in particular that you are trying to monitor with Packetbeat (e.g. dns, tls, http)?

What's the config that you are using? Have you tried changing any of the capture options.

packetbeat.interfaces.device: eth0
packetbeat.interfaces.snaplen: 1514
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100

Using a specific interface, rather than any, can cut down on the amount of data being processed by Packetbeat.

Disabling flows or setting a custom BPF filter such that it only captures specific traffic that you are interested in can also help.

[radu.capverde]

Hi Andrew,

Thank you for replying.

Please find the full configuration below. The monitoring is aimed at our commonly used technologies and is the same configuration on all machines.
The logic was that if a technology was not in use (say PostgreSQL) its port would see no traffic therefore the flow would be "killed" automatically within the 1st 60 seconds.

Also, all VMs have only one network interface, but not all are named the same, therefore using ANY was an easier option for us.

At this time we are not using any custom capture options but I will look into your suggestion to see if it improves things.

Please let me know of you spot any anomalies in our config that might contribute to the CPU usage.

  #============================== Network device ================================
  
  # Select the network interface to sniff the data. On Linux, you can use the
  # "any" keyword to sniff on all connected interfaces.
  packetbeat.interfaces.device: any
  
  #================================== Flows =====================================
  
  # Set `enabled: false` or comment out all options to disable flows reporting.
  packetbeat.flows:
    # Set network flow timeout. Flow is killed if no packet is received before being
    # timed out.
    timeout: 60s
  
    # Configure reporting period. If set to -1, only killed flows will be reported
    period: 60s
  
  #========================== Transaction protocols =============================
  
  packetbeat.protocols.icmp:
    # Enable ICMPv4 and ICMPv6 monitoring. Default: false
    enabled: true
  
  packetbeat.protocols.dns:
    # Configure the ports where to listen for DNS traffic. You can disable
    # the DNS protocol by commenting out the list of ports.
    ports: [53]
  
    # include_authorities controls whether or not the dns.authorities field
    # (authority resource records) is added to messages.
    include_authorities: true
  
    # include_additionals controls whether or not the dns.additionals field
    # (additional resource records) is added to messages.
    include_additionals: true
  
  packetbeat.protocols.http:
    # Configure the ports where to listen for HTTP traffic. You can disable
    # the HTTP protocol by commenting out the list of ports.
    ports: [80, 8080, 8000, 5000, 8002]
  
  packetbeat.protocols.mysql:
    # Configure the ports where to listen for MySQL traffic. You can disable
    # the MySQL protocol by commenting out the list of ports.
    ports: [3306]
  
  packetbeat.protocols.pgsql:
    # Configure the ports where to listen for Pgsql traffic. You can disable
    # the Pgsql protocol by commenting out the list of ports.
    ports: [5432]
  
  packetbeat.protocols.oracle:
    # Configure the ports where to listen for Redis traffic. You can disable
    # the Redis protocol by commenting out the list of ports.
    ports: [1521]
  
  packetbeat.protocols.https:
    # Configure the ports where to listen for Thrift-RPC traffic. You can disable
    # the Thrift-RPC protocol by commenting out the list of ports.
    ports: [443]
  
  packetbeat.protocols.rsyslog:
    # Configure the ports where to listen for MongoDB traffic. You can disable
    # the MongoDB protocol by commenting out the list of ports.
    ports: [514]
  
  packetbeat.protocols.nfs:
    # Configure the ports where to listen for NFS traffic. You can disable
    # the NFS protocol by commenting out the list of ports.
    ports: [2049]

[steffens (Steffen Siering)]

Can you also show the other settings (e.g. output) ?