Pipeline help -

[Carlos_Magalhaes]

Hi all,

Really need some help been trying to get this pipeline up and running but I just can't see what I am doing wrong. Thank you to @Christian_Dahlqvist for pointing me to : the new feature in 6.5.1 which allows you to send information to distributedpipelines using upstream and downstream pipelines.

I have used the example on the link above but just changed some of the info to reflect my configuration. At first with the normal logger set to debug all I could see was:

[2018-11-28T19:10:35,494][DEBUG][logstash.runner          ] --------------- Logstash Settings -------------------
[2018-11-28T19:10:35,552][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations 
from YAML {:location=>"/etc/logstash/pipelines.yml"}
[2018-11-28T19:10:35,609][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash 
stopped processing because of an error: (SystemExit) exit

No other information, so I used journalctl -u logstash.service -f and I got a bit more info:

Nov 29 17:10:53 machine logstash[29813]: [2018-11-29T17:10:53,209][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"} Nov 29 17:10:53 machine logstash[29813]: ERROR: Failed to read pipelines yaml file. Location: /etc/logstash/pipelines.yml

1. I checked the file is there
2. I checked the permissions and it is set correctly

I have run out of ideas put the old config back i.e.

-pipeline.id: mypipeline
 path.config: "/path/to/conf.d/*.conf"

And that works (has same file permissions). I am losing my mind here would really appreciate the help.

- pipeline.id: beats-server
      config.string: |
        input { beats { port => 5044 } }
        output {
            if [type] == suricata {
              pipeline { send_to => suricata }
            } else if [type] == snort {
              pipeline { send_to => snort }
            }
       }
    - pipeline.id: suricata-processing
      config.string: |
        input { pipeline { address => suricata } }
        output {
          elasticsearch {
             hosts => ["localhost"]
             id => "output_elasticsearch_suricata"
             index => "logstash-%{+YYYY.MM.dd}"
           }
        }
    - pipeline.id: snort-processing
      config.string: |
        input { pipeline { address => snort } }
        output {
          elasticsearch {
           id => "output_elasticsearch_snort"
           hosts => ["localhost"]
           index => "snort-%{+YYYY.MM.dd}"
           template => "/etc/logstash/snort/templates/snort.template.json"
           template_name => "snort-1.0.0"
           template_overwrite => "true"
      }
    }