Packetbeat- Duplicated flow records even intermediate report is disabled

Elastic Stack Beats
1

Hi,
I am just curious why we are getting duplicated logs of a flow.

Screen Shot 2563-10-06 at 16.59.59
Screen Shot 2563-10-06 at 16.59.593360×1698 384 KB

As I understand, the flow id should be uniqued and when flow.final is to true, doesn't that mean there should be no more data for this flow?

This is the configuration

# ================================== General ===================================
fields.dc: {{ dc }}
fields_under_root: true

# =============================== Network device ===============================
packetbeat.interfaces.device: any
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.ignore_outgoing: true
packetbeat.interfaces.buffer_size_mb: 100
packetbeat.interfaces.bpf_filter: "tcp port 8500 or tcp port 8300"

# =================================== Flows ====================================
packetbeat.flows:
  timeout: 30s
  period: -1s

# =================================== Internal queue ====================================
# queue:
#   mem:
#     events: 4096
#     flush.min_events: 2048
#     flush.timeout: 0s

# =========================== Transaction protocols ============================
packetbeat.protocols:
  - type: http
    ports: [8500]

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  enabled: true
  hosts: {{elasticsearch_hosts}}
  index: "unsafe-it-devops-services-consul-packetbeat-{{dc}}-%{+yyyy.MM.dd}"
  #bulk_max_size: 100

# ======================= Elasticsearch template setting =======================
setup.template.name: "unsafe-it-devops-services-consul-packetbeat"
setup.template.pattern: "unsafe-it-devops-services-consul-packetbeat-*"
setup.template.settings:
    index.number_of_shards: 1

# ====================== Index Lifecycle Management (ILM) ======================
setup.ilm.enabled: false

# ================================== Logging ===================================
logging.level: info
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/packetbeat
  name: packetbeat
  keepfiles: 10
  permissions: 0644

# ================================= Processors =================================
processors:
   - include_fields:
      fields: ["dc", "destination.ip", "flow.final", "flow.id", "destination.port", "http.request.method", "http.response.status_code", "source.ip", "url.path" ]
   - drop_event:
        when: 
          or:
            - and:
              - not.equals:
                  destination.port: 8300
              - not.equals:
                  destination.port: 8500
            - and: 
              - equals:
                  type: flow
              - equals:
                  destination.port: 8500
            - and:
              - equals:
                  typs: flow
              - equals:
                  destination.port: 8300
              - not.equals:
                  destination.ip: "{{ ansible_default_ipv4.address }}"

I put flow.period to -1s to disable intermediate report

Best regards,
Wathanyu

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.