Multiple Logstash pipelines

Elastic Stack Logstash
1

Hi,

I am using Logstash 6.5 as a service on a windows machine. I have defined a pipelines.yml file that looks like this:
# List of pipelines to be loaded by Logstash
- pipeline.id: filebeat
path.config: "CPA/LogStash/config\logstash-filebeat.conf"

# List of pipelines to be loaded by Logstash
 - pipeline.id: K5DLWDBSSIT2909
   path.config: "CPA/LogStash/config/logstash-K5DLWDBSSIT2909.conf"

When i run logstash from cmd whitout a parameter like '-f' or '-e' the output is like thiis:
"Cannot read the pipelines file"

Does anyone have any ideea how can i resolve this?

Thanks!

2

Is your pipelines.yml file in the directory you have set in path.settings?

3

path.settings is located in logstash.yml?
Pipelines.yml is located in the same folder with logstash.yml

4

Unless you override it on the command line, paths.settings would be set in logstash.yml, yes.

5

The config files that i defined in the path.config must be in the same place where logstash is installed? I don't override in the command line and is still doesn't work..

6

The error is now "Failed to read pipeline yaml file" It matters on which drive is logstash installed? Does it have to be C drive?

7

Can you post the entire output from logstash when you run it from the command line. Please post it as text, select it, and click on </> in the toolbar above the edit panel. You should then see it displayed in the preview panel on the right like this

...
8

C:\Users\zDL-ChERP_GSTS_setup>E:\TotalSoft\CPA\LogStash\bin\logstash.bat
Sending Logstash logs to K:/LogStash/Log which is now configured via log4j2.prop
erties
ERROR: Failed to read pipelines yaml file. Location: E:/TotalSoft/CPA/LogStash/c
onfig/pipelines.yml
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAM
E.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace
] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[2019-05-16T18:04:17,952][ERROR][org.logstash.Logstash ] java.lang.IllegalSta
teException: Logstash stopped processing because of an error: (SystemExit) exit

9

Does that file exist?

10

Yes!

11

If you cd to the E: drive and run the same command do you get the same error?

12

Yes, i get the same error. I don't know what else to try.

13

Since it seems to be on Windows, put backslashes "" and not slashes ? (just guessing, never used Elasticsearch on Windows).

14

Hi,

Now i get a different error : error logstash.config.sourceloader no configuration found in the configured sources.

Any help?
Thanks!

15

You changed slashes to backslashes inside the pipelines.yml too ? ( ```
path.config: "CPA/LogStash/config/logstash-K5DLWDBSSIT2909.conf"

16

Hi,

I try with slashes and with backslashes and it still doesn't work...I don't know what else to try.
If i run just a conf file without using the pipelines.yml it's all good.

Any suggestions?
Thanks!

17

SOLVED!!

I moved the config files into a directory conf.d and after that logstash began to read from pipelines.yml

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.