Packet loss in af_packetbeat mode

Elastic Stack Beats
1
  • Packetbeat Version: 6.2.4
  • Operating System: ubuntu16.04

Hello, I found packetbeat lost packets when capture mirrored HTTP traffic.The way I found is that I send 100 packets every 10 minutes, and then count how many packets I receive to see if there is a packet loss.
Packetbeat runs in af_packet mode. Mirror traffic is about 400 bit/s. I find that packetbeat sometimes loses more than 10% packets, and even loses 80% in severe cases.

Please tell me if this is normal. How can I improve the packet loss rate?

There is something you need to know:

  • Shutting down gro doesn't help
  • There is only TCP traffic in mirror traffic
  • There are enough CPU and memory resources, but they are not fully utilized.
  • There was no error in the packetbeat log.

Here is my config:

#============================== Network device ================================
packetbeat.interfaces.device: eth0
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 3072
#========================== Transaction protocols =============================
packetbeat.protocols:
- type: http
  ports: [80, 7001, 8080, 8000, 8001, 8081, 5000, 5001, 8002, 11080, 82, 7002, 8088, 18880]
  send_all_headers: true
  include_body_for: ["text/xml", "text/html", "text/plain", "application/xml", "application/json", "multipart/form-data", "application/csv", "text/csv", "application/x-ndjson", "application/x-amf"]
  split_cookie: true
  max_message_size: 548500
#================================ General =====================================
processors:
- drop_event:
    when:
       not:
           or:
                - regexp:
                      ip: "10.xxx.*.*"
                - regexp:
                      ip: "10.xxx.*.*"
#------------------------------- Kafka output ----------------------------------
output.kafka:
  enabled: true
  hosts: ["10.130.97.27:9093", "10.130.97.28:9093", "10.130.97.29:9093"]
  topic: quicksand_corp

  metadata:
    # Max metadata request retry attempts when cluster is in middle of leader
    # election. Defaults to 3 retries.
    #retry.max: 3

    # Waiting time between retries during leader elections. Default is 250ms.
    #retry.backoff: 250ms

    # Refresh metadata interval. Defaults to every 10 minutes.
    refresh_frequency: 10h

  worker: 6
  bulk_max_size: 10240

  # The number of messages buffered for each Kafka broker. The default is 256.
  channel_buffer_size: 10240
  max_message_bytes: 8000000

#------------------------------- Syslog output -----------------------------------
logging.to_files: true
logging.files:
  path: /var/log/packetbeat/corp
  name: packetbeat
2

Hi,

The traffic is ridiculously low to cause any troubles.

A few questions:

  • How do you measure this packet loss?
  • Can you share:
    • A packet capture (pcap) with a sample of this kind of traffic.
    • The debug log of packetbeat while exhibiting this problem (run with -d '*')
3

Two thoughts...

  1. Check to see if you have disabled TCP offload engine - if you google how to do this in Linux there’s various ways. I’ve recently found packbeat telling me it’s lost packets.

  2. Run tcpdump as a second independent packet capture tool and compare your pcaps - they should both show the same thing.

4

I have already disabled TCP offload, but it doesn't help.

5

Hi,

I send 100 HTTP requests every 10 minutes and then check how many requests I receive on elastic search to determine how many packets are lost.

Because it is the internal flow of the enterprise, which contains sensitive data, it is inconvenient to provide pcap files.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.