Pipelines.yml ignored on logstash

Elastic Stack Logstash
1

I have looked at the following:

I have LS 8.3.3 and here is version 1 of pipelines.yml located in /opt/logstash/config/pipelines.yml

- pipeline.id: main
  path.config: "/opt/logstash/lab_conf/*.conf"
- pipeline.id: elastic-processing
  path.config: "/opt/logstash/elastic_conf/*.conf"
- pipeline.id: qradar-processing
  path.config: "/opt/logstash/qradar_conf/*.conf"

Here is /opt/logstash/config/logstash.yml

node.name: logstash1
path.data: /opt/logstash/data
#path.config: /opt/logstash/lab_conf

when I run with this config, here is the result:

root@logstash1:/opt/logstash# bin/logstash -t
Using bundled JDK: /opt/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /opt/logstash/logs which is now configured via log4j2.properties
[2022-09-20T19:25:27,163][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/logstash/config/log4j2.properties
[2022-09-20T19:25:27,167][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.3.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-09-20T19:25:27,169][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
ERROR: Failed to read pipelines yaml file. Location: /opt/logstash/config/pipelines.yml
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help
[2022-09-20T19:25:27,389][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
	at opt.logstash.lib.bootstrap.environment.<main>(/opt/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

If I uncomment the path.config in logstash.yml, it still doesn't read the pipelines.yml file:

root@logstash1:/opt/logstash# bin/logstash -t
Using bundled JDK: /opt/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /opt/logstash/logs which is now configured via log4j2.properties
[2022-09-20T19:38:01,975][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/logstash/config/log4j2.properties
[2022-09-20T19:38:01,979][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.3.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-09-20T19:38:01,981][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-09-20T19:38:02,170][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-09-20T19:38:02,534][INFO ][org.reflections.Reflections] Reflections took 60 ms to scan 1 urls, producing 124 keys and 408 values
[2022-09-20T19:38:02,828][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
Configuration OK
[2022-09-20T19:38:02,829][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

I have also tried naming the first pipeline something else besides 'main' and that doesn't work either.

Suggestions?

2

Version 2 of the pipelines.yml just moves all the config files into the pipelines.yml file using config.string.

For example:

- pipeline.id: main
  config.string: |
    input { tcp { port => 1514 }}
    output {
        pipeline { send_to => elastic }
        pipeline { send_to => qradar }
    }

Which just makes a long pipelines.yml file but doesn't fix the problem...

3

It is failing to read the file, most likely a permissions error.

That is expected. You can use path.confiig or pipelines.yml, you cannot use both.

4

perms are:

-rw-r--r-- 1 logstash logstash  6414 Sep 20 19:13 pipelines.yml

These are the same perms for all the files in the /opt/logstash/config folder.

There were root.root, but I was trying different things to get it to work..... should I change them back?
(sorry for the noob question....)

This was installed using the tarball... not rpm or deb...

5

Does the logstash user has permissions to the config path as well?

If you want to use pipelines.yml, remove the paht.config line from logstash.yml.

Also, try to use --path.settings in the command line pointing to the folder where the logstash.yml and pipelines.yml file are stored.

bin/logstash --path.settings /opt/logstash/config should start logstash using the pipelines.yml.

6

Will check 1st thing tomorrow....

7

OK, I removed the path.config from the logstash.yml

su to logstash, checked permissions. As logstash, I could read logstash.yml and pipelines.yml

exited to root, then tried to run /opt/logstash/bin/logstash as root. same error.

tried to run as root using /opt/logstash/bin/logstash --path.settings /opt/logstash/config. same error.

chowned entire /opt/logstash directory to logstash.logstash and tried to run as root. same error.

su to logstash and tried to run as logstash. same error.

Still able to read both files as logstash user.

8

OK. I just removed the pipelines.yml file from the config directory, then su'd to the logstash user and created a new pipelines.yml file and started logstash as the logstash user. That worked.

from the log:

...snip...
Pipelines running {:count=>3, :running_pipelines=>[:main, :"elastic-processing", :"qradar-processing"], :non_running_pipelines=>}
...snip...

(I got a bunch of pipeline errors, but that is a different problem I think)

Works when I run as root as well.

9

This could mean that your old pipelines.yml had some formatting error, which is one thing that could give you the error you were getting before.

I would say that you should avoid running logstash as root even for testing, when logstash runs it creates some temporary files and paths depending on your pipeline, if you run it as a root, it will create them as the root user, which could give you permissions errors when you run as the logstash user.

10

Got it. Makes sense why I should not run as root. (besides other good reasons)

Thanks for the help.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.