Filebeat & ingest pipelines

Hi there,
is it correct to say that you only need to load an ingest pipeline (e.g. filebeat setup --pipelines --modules system,nginx,mysql) when you are using elasticsearch ingest node as output? Or do also need to do this if you are only using the logstash outpt?
Thanks and kind regards!

No, you don't need ingest pipelines unless you are going to use them :slight_smile:

Technically, logstash can direct output to an Elasticsearch pipeline with the "pipeline" parameter, so this reads a little like a trick exam question :slight_smile:

Hi there,

thanks for the answer. To what logstash pipeline parameter are you referring to?
(https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html)

Would you mind give me a quick example?
Thank you very much and kind regards
Stefano (Metronom - METRO IT)

The "pipeline =>" in the example below. This is an ansible template, so {{ fields }} aren't filled in at this point. This is how I send filebeat module events that need ingest pipeline processing from logstash to elastic in the same pipe as events that don't need ingest processing.

....
  else if "use_ingest" in [tags] and [fileset][module] {
    elasticsearch {
      hosts => [{{ ES_http }}]
      cacert => "/..../certs/https_interm.cer"
      user => "{{ elastic.user }}"
      password => "{{elastic.pass }}"
      sniffing => false
      manage_template => false
      pipeline => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fileset][module]}-%{[fileset][name]}-pipeline"
      ilm_enabled => true
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fields][app_id]}-%{[fields][campus]}"
    }
....

Hi @StefanoM and @rugenl,

There are two separate things to think about here:

  1. Loading ingest pipelines, and
  2. Referencing the loaded ingest pipelines.

This documentation page explains the difference between these two in detail: https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html#use-ingest-pipelines.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.