ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml

private proprietary enterprise professional background developing on Open Source.. what a pain in the neck!

new into this nightmare, ive spent around 36 hrs fighting new ELK install ( from ubuntu ISO to application ) Setup from scratch ELK installed on ubuntu 1804 - had lots of issues on the way until get to the below error: mainly related to Logstash permissions, settings etg. I have read and learn on the way so much but I need help troubleshooting this:
netstat -plnt

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 862/node
tcp 0 0 127.0.0.1:9200 0.0.0.0:* LISTEN 965/java
tcp 0 0 127.0.0.1:9300 0.0.0.0:* LISTEN 965/java
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 801/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 950/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 22962/cupsd
tcp6 0 0 :::22 :::* LISTEN 950/sshd
udp 0 0 0.0.0.0:39582 0.0.0.0:* 824/avahi-daemon: r
udp 0 0 127.0.0.53:53 0.0.0.0:* 801/systemd-resolve
udp 0 0 0.0.0.0:631 0.0.0.0:* 22963/cups-browsed
udp 0 0 0.0.0.0:5353 0.0.0.0:* 824/avahi-daemon: r
udp6 0 0 :::48471 :::* 824/avahi-daemon: r
udp6 0 0 :::5353 :::* 824/avahi-daemon: r

Who the hell is avahi-daemon?
root@adminri-virtual-machine:/usr/share/logstash/bin# sudo /usr/share/logstash/bin/logstash
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[ERROR] 2020-04-23 00:40:35.356 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
root@adminri-virtual-machine:/usr/share/logstash/bin#

I dont really understand why sometimes I run logstash it gets run from different paths: /etc/logstash/ . It seems the reason resides when I first downloaded and installed the package it didnt come from elastic but other repo instead mixing and messing the execution, configurational, settings paths.
A mess.
It did happen the same on FreeBSD where I run my prod pfsense and installed Beats for Freebsd- I had to be very careful from where do I run the service ( path ) .

Anyway
run

root@adminri-virtual-machine:/usr/share/logstash/bin# sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/ --path.settings=/etc/logstash --log.level=debug

[2020-04-23T01:00:53,816][DEBUG][logstash.runner ] --------------- Logstash Settings -------------------
[2020-04-23T01:00:53,868][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-23T01:00:53,884][FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[2020-04-23T01:00:53,893][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

And another observation copying and paste from to terminal modify certain characters, a missing 'double quote' here, extra # there ... 0 consistency between applications I dont see where the efficiencies are

thanks to who ever picks up this thread.. Im new to the tech so potentially a tech controbutor.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.