Ingest Pipeline - ignore_failure is ignored

Elastic Stack Beats
1

Hi,

We are attempting to ingest PANW events via filebeat->logstash->elasticsearch.

The filebeat-8.4.0-panw-panos-pipeline fails due to the second processor being:

rename "message" to "event.original"

Our event already has event.original, so the processor fails and the pipeline processing stops. Adding ignore_failure to that processor makes no difference, it still fails on it and stops processing.

If we remove "event.original" in logstash, we get no events ingested.

If we remove the processor altogether, events are not ingested (or at least we can't find them in the expected index. There are no error messages in filebeat or logstash.

Has anyone come across this before, or does anyone have any ideas?

Thanks

2

The PANW pipeline in filebeat assumes that you will send it directly to Elasticsearch, without Logstash in the middle, this is the same for all filebeat modules.

Some modules will parse the message using an ingest pipeline in Elasticsearch, others, like the PANW module, will parse the message directly in Filebeat, so the filebeat-8.4.0-panw-panos-pipeline expects an already parsed message.

Adding Logstash between Filebeat and Elasticsearch may break things since Logstash will add some fields and depending if you have any filters it may also override other fields.

Are you using the module in Filebeat or are you using a custom input to read/receive your logs from your Palo Alto? Can you add an stdout output to your logstash pipeline and share the output you are getting from Logstash? Any reason to use Logstash in this case or can you ditch it?

Also, If I'm not wrong Logstash will only add the event.original field if you have ecs_compatibility enabled for the pipeline, you may try to set it to disabled and see if it works.

Just set it in the pipeline configuration in your pipelines.yml file.

pipeline.ecs_compatibility: disabled
3

@leandrojmp thanks for your reply.

I have attempted to send directly from filebeat using the panw module, but nothing is indexed to elasticsearch. Neither filebeat nor elasticsearch logs any errors.

If I send via logstash, I can see the events (via stdout) and the events are indexed in elasticsearch, however the ingest pipelines fail to complete due to event.original existing in the event. This then means that the PANW dashboards don't work as the pipeline should add some fields required by the dashboards.

Setting pipeline.ecs_compatibilty made no difference at all.

If I could add ignore_failure to that pipeline processor then I believe the pipeline processing would complete, however adding ignore_failure does absolutely nothing as it still stops processing. Frustrating :frowning:

4

Perhaps share your logstash config....in it's entirety.

Also assuming you ran initially
filebeat setup -e

Also, I always suggest running directly from filebeat to elasticsearch first before putting logstash in the middle.

Make sure everything works and then put logstash in the middle.

Your logstash config can make a difference.

5

Thanks @stephenb

Yes, I have run filebeat -e setup a number of times. Each time I attempt to edit the ingest pipeline, for example removing the processor in question and saving, nothing works and I need to setup the pipelines again.

I can't even re-order the processors in the pipeline, they just stay in the same order no matter what I do.

My logstash config is as follows:

input {
  beats {
    port => 55555
  }
}

filter {
    if [event][created] {
    date {
      match => [ "[event][created]", "YYYY/MM/dd HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
      target => "[event][created]"
    }
  }
  if [event][start] {
    date {
      match => [ "[event][start]", "YYYY/MM/dd HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
      target => "[event][start]"
    }
  }
}

output {
  stdout {
    codec => rubydebug { metadata => true }
  }

  if [@metadata][beat] and [@metadata][pipeline] {
    elasticsearch {
      hosts => ["https://XXXXXXXX:9200"]
      user => "XXXXXXXX"
      password => "XXXXXX"
      ssl => true
      cacert => "/etc/logstash/certs/http_ca.crt"
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}"
      ilm_enabled => true
      action => create
    }
  }
  else if [@metadata][beat] {
    elasticsearch {
      hosts => ["https://XXXXXXXXX:9200"]
      user => "XXXXXXXX"
      password => "XXXXXXXX"
      ssl => true
      cacert => "/etc/logstash/certs/http_ca.crt"
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      ilm_enabled => true
      action => create
    }
  }
}

The mutates are due to the requirement for milliseconds in the date fields otherwise the events aren't ingested.

Thanks.

6

This should work, if this isn't work you may have some issue in the format of your data.

What is your source? Are you reading log files or receiving the logs via TCP/UDP?

Can you share the raw data you are receiving in Filebeat? What is your Filebeat configuration?

Can you share it? You need to show what is the output from Logstash.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.