Failed to read pipelines yaml file

[wikedeye (Conan)]

I am sorry if this is an easy problem to fix. I have been trying to figure out what is wrong for a while and unfortunately I do not have enough experience with linux to understand some of what I see as solutions. I am trying to run elastiflow in my ELK stack and I cannot get my pipeline configuration to work. I have tried asking the creator of elastiflow, but he seems to think it is an issue with logstash. Here is a debug output from my server.

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2019-03-04 16:00:03.461 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2019-03-04 16:00:03.474 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help
[ERROR] 2019-03-04 16:00:04.138 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

My pipeline.yml file is located at /ect/logstash/pipelines.yml

# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"

I know this message is supposed to tell me what is wrong, but again I do not have much experience with linux and I am unable to understand what I need to do. If more information is needed, please let me know what and I will provide it.

[Badger]

How are you running logstash?

The default for --path.settings is /usr/share/logstash/config, so that is where it is looking for pipelines.yml, logstash.yml, and log4j2.properties.

You probably want to use "--path.settings /etc/logstash" on the command line.

[wikedeye (Conan)]

Sorry if I answer this wrong. I used systemctl to run logstash.

When I first set this up there was a default pipeline called main in the same file as the one I posted. I deleted main because it is supposedly not needed according to the guide I used.

I have seen the method of - - path.settings, but I don't really know what the full command would be to correctly set the pipeline path. Also /etc/logstash is where the pipeline.yml, logstash.yml, and log4jz currently is.

Would this command have caused logstash look in the wrong directory for the conf file?
sudo /usr/share/logstash/bin/system-install

[Badger]

It is OK to replace the "main" pipeline with something else.

The problem is that logstash expects logstash.yml, pipelines.yml, and log4j2.properties to be in /usr/share/logstash/config and they are actually in /etc/logstash. The way to fix that is to pass "--path.settings /etc/logstash" to logstash on the command line.

What you have now is a systemd question, not a logstash question. On my system I have an /etc/systemd/service/logstash.service which, if it were ever called, would invoke logstash using

ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"

However, the services reside in /lib/systemd/service so that has no impact on my server. You need to explore what units systemd has configured, where they are configured, and what the systemd configuration file for logstash looks like.

I certainly did not write this, so I assume the default logstash install adds /etc/systemd/system/logstash.service that contains

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

It might be as simple as copying that file to /lib/systemd/system.

[wikedeye (Conan)]

Ok thanks. I will look a little deeper tomorrow when I get back to the office.