Pipelines.yml failed to read with Pipeline to Pipeline model

Elastic Stack Logstash
1

I had previously multiple pipelines and i added Pipeline to Pipeline model in my pipelines.yml
But since then i get this error:

ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help

So here is my pipeline.yml file:

- pipeline.id: update-pipeline
  path.config: "/usr/share/logstash/pipeline/update.conf"
  pipeline.workers: 1

- pipeline.id: enrichment-pipeline
  path.config: "/usr/share/logstash/pipeline/enrichment.conf"

- pipeline.id: skoupa
  config.string: |
    input { beats { port => 5044 } }
    output { pipeline { send_to => ["to-es", "enrich"] } }

- pipeline.id: straight-es
  config.string: |
    input { pipeline { address => "to-es" } }
    output { elasticsearch { hosts => ["https://localhost:9200"] 
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        user => logstash_internal
        password => apass
        ssl => true
        cacert => "/usr/share/logstash/config/elasticsearch-ca.pem"
        http_compression => true
	 	    sniffing => false } }
2

It is failing to read the file. That has nothing to do with the contents of the file, since logstash has not seen the contents. The most likely explanations are that the file does not exist or the user running logstash does not have permissions to access the file or one of the directories in the path.

3

Commented the config and added it to separate files and now it is working. It has nothing to do with permissions or wrong path

- pipeline.id: misp-update-pipeline
  path.config: "/usr/share/logstash/pipeline/misp.conf"
  pipeline.workers: 1

- pipeline.id: misp-enrichment-pipeline
  path.config: "/usr/share/logstash/pipeline/enrichment.conf"

- pipeline.id: skoupa
  path.config: "/usr/share/logstash/pipeline/skoupa.conf"

  # config.string: |
  #   input { beats { port => 5044 } }
  #   output { pipeline { send_to => ["to-es", "enrich"] } }

- pipeline.id: straight-es
  path.config: "/usr/share/logstash/pipeline/straight_es.conf"
  
  # config.string: |
  #   input { pipeline { address => "to-es" } }
  #   output { elasticsearch { hosts => ["https://localhost:9200"] 
  #       index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  #       user => logstash_internal
  #       password => logstashpass
  #       ssl => true
  #       cacert => "/usr/share/logstash/config/elasticsearch-ca.pem"
  #       http_compression => true
  #       sniffing => false } }
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.