Multiple filebeat inputs and multiple logstash pipelines

[nino]

Dear all,

this is my scenario:

one directory with two types of files that i want to proccess with one pipeline each. File types are identified by his name. I am very new to pipeline logstash, i usually go with a single logstash configuration but things are getting complex and i would like to use different pipelines for each type of file to separate logic and a better maintenance

filebeat.inputs:

- type: log
  enabled: true
  paths: C:\files\*apache-log*.txt
  tags: ["type1"]

- type: log
  enabled: true
  paths: C:\files\*ngnix-log*.txt
  tags: ["type2"]

How can apply one pipeline base on the file type using multiple pipelines mode?

- pipeline.id: pipeline-for-type1-files
  path.config: "/etc/path/to/type1-pipeline.config"
- pipeline.id: pipeline-for-type2-files
  path.config: "/etc/different/path/type2-pipeline.config"

If logstash recieves type1 files apply pipeline-for-type1-files and if recieves type2 apply pipeline-for-type2-files. Could you give some sample code of how to handle this?

Best regards

[Badger]

Look at the examples for the distributor pattern in the pipeline-to-pipeline communication documentation.

[nino]

Thank you so much badger. Just to simplify a little and while i learn how distributor pattern works, should this simple configuration could work?.

If i understand well how pipeline config files work, if i filter by tag with and if statement in each pipeline config should filter and apply this pipeline only in the events from the file tagged as "type1". Am i correct Badger?

type1-pipeline.config

input {
  
  beats {
    port => "5044"
  }  
  
}

filter {
  if [tags] == "type1" {

  }
}

type2-pipeline.config

 input {
      
      beats {
        port => "5044"
      }  
      
    }

    filter {
      if [tags] == "type2" {

      }
    }

Thanks again

[Badger]

Not quite. Go back and read the documentation. They use config.string rather than config.path, but you could do the same with config.path

- pipeline.id: distributor
  config.string: |
    input { beats { port => 5044 } }
    output {
        if "type1" in [tags] {
          pipeline { send_to => type1 }
        } else if "type2" in [tags] {
          pipeline { send_to => type2 }
        } else {
          pipeline { send_to => fallback }
        }
    }
- pipeline.id: type1
  config.string: |
    input { pipeline { address => type1 } }
    filter {
       # type1 filter statements here...
    }
    output {
      # type1 output here...
    }
- pipeline.id: type2
  config.string: |
    input { pipeline { address => type2 } }
    filter {
       # type2 filter statements here...
    }
    output {
      # type2 output here...
    }
- pipeline.id: fallback
  config.string: |
    input { pipeline { address => fallback } }
    filter {
       # fallback filter statements here, if any
    }
    output {
      # fallback output here...
    }

Having another pipeline for events that are neither type1 nor type2 is not mandatory, you could just make 'pipeline { send_to => type2 }' the 'else' option.

[nino]

Thank you so much @Badger!