Configure Pipelines in pipelines.yml

[cawoodm (Marc)]

I know we can specify pipelines in pipelines.yml but can we configure them as well? Ideally we could have one config file which is flexible enough to do different things depending on fields we send to it from pipelines.yml.

 pipeline.id: my-pipeline_1
  path.config: "/etc/path/to/myflexible.config"
  fields.custom: "value1"

 pipeline.id: my-pipeline_2
  path.config: "/etc/path/to/myflexible.config"
  fields.custom: "value2"

[kharvey (Ken Harvey)]

I'm not sure what you are asking for. You may take a look at this:
https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html

I use if else if logic in my pipelines.yml to send different log files to different pipelines.

- pipeline.id: port5044
  config.string: |
    input { 
      beats { 
        port => 5044
      } 
    }
    output {
	  if "dmesg" in [fields] {
        pipeline {
          send_to => dmesg_log
        }
      }
      else if "postgres" in [fields] {
        pipeline {
          send_to => postgres_log
        }
      }
      else if "meraki" in [fields] {
        pipeline {
          send_to => meraki_syslog
        }
      }
      else if "forum" in [fields] {
        pipeline {
          send_to => forum_help
        }
      }
      else {
        pipeline {
          send_to => zip_file
        }
      }
    }
- pipeline.id: dmesg
  path.config: "/etc/logstash/conf.d/dmesg/*"
- pipeline.id: postgres
  path.config: "/etc/logstash/conf.d/postgres/*"
- pipeline.id: forum_help
  path.config: "/etc/logstash/conf.d/ForumHelp/*"
- pipeline.id: meraki
  path.config: "/etc/logstash/conf.d/meraki/*"
- pipeline.id: zips
  path.config: "/etc/logstash/conf.d/zip_file.conf"

You can use config.string to do different things like add fields and such if you want.