Found Duplicated HTTP Events in Kubernetes Cluster

Elastic Stack Beats
1

Hi,
We're using Packetbeat to capture http traffic in Kubernetes Cluster. We've observed that there are duplicated http events even set ignore_outgoing to true.

Example as blow:

image
image3078×400 72.6 KB

  • same network.community_id, source.ip, source.port, destination.ip, destination.port
  • the difference between these two event is agent.hostname, hostname and node.name
  • the actually event is the one with kubernetes.pod.uid attached

My question is if there's any sort of method to avoid duplicated events?

2

the configuration:

packetbeat.interfaces.device: any
packetbeat.interfaces.type: af_packet

packetbeat.ignore_outgoing: true

packetbeat.protocols:
- type: http
  enable: true
  send_all_headers: true
  split_cookie: true
  include_body_for: ["application/json", "application/x-www-form-urlencoded"]

packetbeat.flows:
  timeout: 30s
  period: 10s

processors:
  - add_cloud_metadata:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      indexers:
      - ip_port:
      matchers:
      - field_format:
          format: '%{[server.ip]}:%{[server.port]}'