Elastic Search is receiving data from beat, but no documents appear

DavidTurner · February 7, 2020, 10:52am

I've looked through the source and I do not see any obvious additional logging that can be enabled to trace a single document through an ingest pipeline, sorry.

Although I'm not familiar with how Beats configures Elasticsearch, I had a quick look at the source and I see a single drop processor here:

github.com

elastic/beats/blob/v7.5.2/x-pack/filebeat/module/panw/panos/ingest/pipeline.yml#L176-L187


 # Set event.category depending on log type.
- set:
    field: event.category
    value: network_traffic
    if: 'ctx?._temp_?.message_type == "TRAFFIC"'
- set:
    field: event.category
    value: security_threat
    if: 'ctx?._temp_?.message_type == "THREAT"'


- drop:
    if: 'ctx?.event?.category == null'

Is this the pipeline you're running? If so, it looks like it drops messages that do not have _temp_.message_type set to TRAFFIC or THREAT and in the request above this field is SYSTEM.

Topic		Replies	Views
Filebeat is running on two different systems with two different modules but no documents appear in filebeat indx Beats	2	466	March 6, 2020
Using filebeat syslog input for PANW. Logs not publishing to Kibana Beats filebeat	5	1480	November 26, 2019
Filebeat PANW Module Not Working (continued) Beats filebeat	3	740	April 30, 2020
No data from Filebeat 7.7 - Using Palo Alto module Beats filebeat	2	531	June 19, 2020
Panw module Beats beats-module , filebeat	2	257	November 29, 2021

Elastic Search is receiving data from beat, but no documents appear

Related topics