Filebeat & ingest pipelines

Elastic Stack Beats
1

Hi there,
is it correct to say that you only need to load an ingest pipeline (e.g. filebeat setup --pipelines --modules system,nginx,mysql) when you are using elasticsearch ingest node as output? Or do also need to do this if you are only using the logstash outpt?
Thanks and kind regards!

2

No, you don't need ingest pipelines unless you are going to use them :slight_smile:

Technically, logstash can direct output to an Elasticsearch pipeline with the "pipeline" parameter, so this reads a little like a trick exam question :slight_smile:

3

Hi there,

thanks for the answer. To what logstash pipeline parameter are you referring to?
(https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html)

Would you mind give me a quick example?
Thank you very much and kind regards
Stefano (Metronom - METRO IT)

4

The "pipeline =>" in the example below. This is an ansible template, so {{ fields }} aren't filled in at this point. This is how I send filebeat module events that need ingest pipeline processing from logstash to elastic in the same pipe as events that don't need ingest processing.

....
  else if "use_ingest" in [tags] and [fileset][module] {
    elasticsearch {
      hosts => [{{ ES_http }}]
      cacert => "/..../certs/https_interm.cer"
      user => "{{ elastic.user }}"
      password => "{{elastic.pass }}"
      sniffing => false
      manage_template => false
      pipeline => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fileset][module]}-%{[fileset][name]}-pipeline"
      ilm_enabled => true
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fields][app_id]}-%{[fields][campus]}"
    }
....
5

Hi @StefanoM and @rugenl,

There are two separate things to think about here:

  1. Loading ingest pipelines, and
  2. Referencing the loaded ingest pipelines.

This documentation page explains the difference between these two in detail: https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html#use-ingest-pipelines.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.