Pipelines.yml ignored on logstash

[reswob]

I have looked at the following:

Ignoring the 'pipelines.yml' file because modules or command line options are specified - RPM

i have installed logstash through RPM on my ec2 instance and started < ./logstash --path.settings /etc/logstash/ --path.config /etc/logstash/ /> but i see below error. Can someone help out here? [root@ip-10-1-1-133 bin]# ./logstash --path.settings /etc/logstash/ --path.config /etc/logstash/ -e OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N …

Logtash | Multiple Pipelines

Logstash v6 beta with multiple pipelines configuration

Note, in above output there was only one pipeline started for two config files which are located in /etc/logstash/conf.d/ dir. pipelines.yml seems to have been ignored.

Pipelines.yml ignored on logstash running as a service

First, I saw and tried the solution mentioned here and commented out path.config from /etc/logstash/logstash.yml Looks like when I run logstash from command line all works well, but I need the logstash service to run. ie. "systemctl start logstash" Logstash completely ignores pipelines.yml Any advice or ideas? Best, JD

I have LS 8.3.3 and here is version 1 of pipelines.yml located in /opt/logstash/config/pipelines.yml

- pipeline.id: main
  path.config: "/opt/logstash/lab_conf/*.conf"
- pipeline.id: elastic-processing
  path.config: "/opt/logstash/elastic_conf/*.conf"
- pipeline.id: qradar-processing
  path.config: "/opt/logstash/qradar_conf/*.conf"

Here is /opt/logstash/config/logstash.yml

node.name: logstash1
path.data: /opt/logstash/data
#path.config: /opt/logstash/lab_conf

when I run with this config, here is the result:

root@logstash1:/opt/logstash# bin/logstash -t
Using bundled JDK: /opt/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /opt/logstash/logs which is now configured via log4j2.properties
[2022-09-20T19:25:27,163][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/logstash/config/log4j2.properties
[2022-09-20T19:25:27,167][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.3.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-09-20T19:25:27,169][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
ERROR: Failed to read pipelines yaml file. Location: /opt/logstash/config/pipelines.yml
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help
[2022-09-20T19:25:27,389][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
	at opt.logstash.lib.bootstrap.environment.<main>(/opt/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

If I uncomment the path.config in logstash.yml, it still doesn't read the pipelines.yml file:

root@logstash1:/opt/logstash# bin/logstash -t
Using bundled JDK: /opt/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /opt/logstash/logs which is now configured via log4j2.properties
[2022-09-20T19:38:01,975][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/logstash/config/log4j2.properties
[2022-09-20T19:38:01,979][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.3.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-09-20T19:38:01,981][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-09-20T19:38:02,170][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-09-20T19:38:02,534][INFO ][org.reflections.Reflections] Reflections took 60 ms to scan 1 urls, producing 124 keys and 408 values
[2022-09-20T19:38:02,828][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
Configuration OK
[2022-09-20T19:38:02,829][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

I have also tried naming the first pipeline something else besides 'main' and that doesn't work either.

Suggestions?

[reswob]

Version 2 of the pipelines.yml just moves all the config files into the pipelines.yml file using config.string.

For example:

- pipeline.id: main
  config.string: |
    input { tcp { port => 1514 }}
    output {
        pipeline { send_to => elastic }
        pipeline { send_to => qradar }
    }

Which just makes a long pipelines.yml file but doesn't fix the problem...

[Badger]

@reswob

ERROR: Failed to read pipelines yaml file. Location: /opt/logstash/config/pipelines.yml

It is failing to read the file, most likely a permissions error.

If I uncomment the path.config in logstash.yml, it still doesn't read the pipelines.yml file:

That is expected. You can use path.confiig or pipelines.yml, you cannot use both.

[reswob]

perms are:

-rw-r--r-- 1 logstash logstash  6414 Sep 20 19:13 pipelines.yml

These are the same perms for all the files in the /opt/logstash/config folder.

There were root.root, but I was trying different things to get it to work..... should I change them back?
(sorry for the noob question....)

This was installed using the tarball... not rpm or deb...

[leandrojmp (Leandro Pereira)]

Does the logstash user has permissions to the config path as well?

If you want to use pipelines.yml, remove the paht.config line from logstash.yml.

Also, try to use --path.settings in the command line pointing to the folder where the logstash.yml and pipelines.yml file are stored.

bin/logstash --path.settings /opt/logstash/config should start logstash using the pipelines.yml.

[reswob]

Will check 1st thing tomorrow....

[reswob]

OK, I removed the path.config from the logstash.yml

su to logstash, checked permissions. As logstash, I could read logstash.yml and pipelines.yml

exited to root, then tried to run /opt/logstash/bin/logstash as root. same error.

tried to run as root using /opt/logstash/bin/logstash --path.settings /opt/logstash/config. same error.

chowned entire /opt/logstash directory to logstash.logstash and tried to run as root. same error.

su to logstash and tried to run as logstash. same error.

Still able to read both files as logstash user.

[reswob]

OK. I just removed the pipelines.yml file from the config directory, then su'd to the logstash user and created a new pipelines.yml file and started logstash as the logstash user. That worked.

from the log:

...snip...
Pipelines running {:count=>3, :running_pipelines=>[:main, :"elastic-processing", :"qradar-processing"], :non_running_pipelines=>}
...snip...

(I got a bunch of pipeline errors, but that is a different problem I think)

Works when I run as root as well.

[leandrojmp (Leandro Pereira)]

@reswob

created a new pipelines.yml file and started logstash as the logstash user.

This could mean that your old pipelines.yml had some formatting error, which is one thing that could give you the error you were getting before.

@reswob

Works when I run as root as well.

I would say that you should avoid running logstash as root even for testing, when logstash runs it creates some temporary files and paths depending on your pipeline, if you run it as a root, it will create them as the root user, which could give you permissions errors when you run as the logstash user.

[reswob]

Got it. Makes sense why I should not run as root. (besides other good reasons)

Thanks for the help.