How to have a filebeat config send some prospectors to a pipeline and send some prospectors to no pipeline

[frr255]

I have a Filebeat config containing multiple prospectors. I want some prospectors to be sent to a pipeline. This is easy.

However, I also want some prospectors not to be sent to any pipeline i.e. skip or avoid any pipelines. What's the easiest way to accomplish this?

Thanks!

[Fei_Yao]

Yes, I'm contemplating the same issue. The one I can think of is to have a pipeline defined doing nothing.

PUT _ingest/pipeline/null
{
"description": "do-nothing",
"processors": []
}

Then, you can put multi-pipeline condition in place like:

- pipeline: null
  when.equals:
    fields.document_type: no-pipeline

Love to know other solutions as well.
Thanks

[steffens (Steffen Siering)]

I think with 5.6 you can define the pipeline in the prospector as well. See pipeline setting.

[frr255]

That's great. Thanks!

This will also work.

filebeat:
  prospectors:
    - paths:
        - /PATH
      input_type: log
      fields:
        application: alpha
    - paths:
        - /PATH
      input_type: log
      fields:
        application: bravo
    - paths:
        - /PATH
      input_type: log
      fields:
        application: bravo

output.elasticsearch:
  hosts: ["URL"]
  pipelines:
    - pipeline: bravo
      when.equals:
        fields.application: "bravo"

[steffens (Steffen Siering)]

One can also make use of format strings:

filebeat.prospectors:
- input_type: log
  paths:
    - /PATH
  fields.application: alpha
- input_type: log
  paths:
    - /PATH
  fields.application: bravo

output.elasticsearch:
  hosts: ["URL"]
  pipeline: '%{[fields.application]}'

If fields.application is missing, pipeline will be empty -> event is not send to any pipeline.