Elastic Search is receiving data from beat, but no documents appear

DavidTurner · February 7, 2020, 10:52am

I've looked through the source and I do not see any obvious additional logging that can be enabled to trace a single document through an ingest pipeline, sorry.

Although I'm not familiar with how Beats configures Elasticsearch, I had a quick look at the source and I see a single drop processor here:

github.com

elastic/beats/blob/v7.5.2/x-pack/filebeat/module/panw/panos/ingest/pipeline.yml#L176-L187


 # Set event.category depending on log type.
- set:
    field: event.category
    value: network_traffic
    if: 'ctx?._temp_?.message_type == "TRAFFIC"'
- set:
    field: event.category
    value: security_threat
    if: 'ctx?._temp_?.message_type == "THREAT"'


- drop:
    if: 'ctx?.event?.category == null'

Is this the pipeline you're running? If so, it looks like it drops messages that do not have _temp_.message_type set to TRAFFIC or THREAT and in the request above this field is SYSTEM.