We're trying to use the multiple pipelines feature described here in Logstash/ES 6.5 on Windows but LogStash can apparently not see/find/read the .conf files.
We're starting logstash.bat with no parameters and our pipelines.yml looks like this:
# Pipeline for FileBeats: Application console and access logs
- pipeline.id: filebeat
path.config: "/elk/logstash/config/pipeline.filebeat.conf"
# Pipeline for other application logs
- pipeline.id: other.logs
path.config: "pipeline.other.logs.conf"
The error is:
[2018-12-18T10:43:43,542][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>LogStash::ConfigManagement::ElasticsearchSource::RemoteConfigError, :message=>"Empty configuration for pipeline_id: filebeat", :backtrace=>["D:/elk/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.2-java/lib/config_management/elasticsearch_source.rb:101:
in `get_pipeline'", "D:/elk/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.2-java/lib/config_management/elasticsearch_source.rb:87:
...
We have tried every variation of path.config: "pipeline.filebeat.conf" or path.config: "config\pipeline.filebeat.conf" etc.
It seems LogStash cannot find these .conf files but they do indeed exist in the config folder.
Adding --debug has not helped in seeing where LogStash expects these files.
BONUS QUESTION: How do the pipelines defined in pipeline.yml relate to those defined in logstash.yml in xpack.management.pipeline.id and those created in Kibana??