Pipeline.yml not working

[davehs30 (David Smith)]

on logstash 6.1.2 , I have commented out the path.config in logstash.yml as well as remove the --path-settings from the systemd service file. I get no error messages in the log, even when I have no pipelines.yml files in /etc/logstash. I have tried to put garbage in the pipelines.yml file and still nothing in the logs.

2018-01-25T15:59:46,376][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
[2018-01-25T15:59:58,328][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}

Just repeats... How does one get the pipelines.yml to work?

[guyboertje]

A continual repeat of [logstash.modules.scaffold] Initializing module indicates that the logstash service is stopping with a serious error and the systemd service manager restarts it.

Which operating system are you using?
The actual error message is not logged to LS logging but to the systemd manager logs OR dmesg etc.
You need to find those errors messages - they will tell you more.

On my old Ubuntu box I found them in:
DMESG

[12744.314629] init: logstash main process (7388) terminated with status 1
[12744.314656] init: logstash main process ended, respawning
[12757.405301] init: logstash main process (7442) terminated with status 1
[12757.405313] init: logstash main process ended, respawning
[12770.408035] init: logstash main process (7487) terminated with status 1
[12770.408062] init: logstash main process ended, respawning
[12772.551418] init: logstash main process (7529) terminated with status 143

/var/log/upstart/logstash.log

Sending Logstash's logs to /var/log/logstash which is now configured via
log4j2.properties
ERROR: You cannot use -f since Elasticsearch is configured as the config
store.
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help

[davehs30 (David Smith)]

THANKS!

Yes, seeing that it isn't starting from the syslog file:

Jan 29 13:21:26 logstash logstash[24488]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Jan 29 13:21:27 logstash logstash[24488]: ERROR: Failed to read pipelines yaml file. Location: /etc/logstash/pipelines.yml
Jan 29 13:21:27 logstash logstash[24488]: usage:
Jan 29 13:21:27 logstash logstash[24488]: bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
Jan 29 13:21:27 logstash logstash[24488]: bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
Jan 29 13:21:27 logstash logstash[24488]: bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]

The file is there:
root@logstash:/etc/logstash# ls -la /etc/logstash/pipelines.yml
-rw-r--r-- 1 root root 186 Jan 29 13:20 /etc/logstash/pipelines.yml

Contents of the file:
pipeline.id: netscaler
path.config: "/etc/logstash/btg-pipelines/netscaler.pipeline.conf"
pipeline.id: scaleio
path.config: "/etc/logstash/btg-pipelines/scaleio.pipeline.conf"

I have tried changing the owner of the pipelines.conf to logstash, chmod 777, but still the same error.
I have tried some YAML formating, such as a three --- to start the file.. - infront of the pipeline.id, 5 spaces for indent.. still the same error.

Thanks again for your assistance!!!
When using the path.config in logstash.yml both files are read and processed.

[davehs30 (David Smith)]

OK.. found another post about the pipelines.yml file format.. I had more than 2 spaces in front of the path.config. I reduced it to the 2 spaces and it is now working. Thanks!