How to configure multi-pipeline configuration in logstash 8.6.1?

Elastic Stack Logstash
1
  1. Basic Information
logstash version: 8.6.1
Logstash installation method: tar.gz
logstash installation directory: /opt
OS: CentOS 7
  1. ogstash.yml configuration file
node.name: logstash01
path.data: /data/logstash01
pipeline.ordered: auto
path.config: /opt/logstash/config/*.conf
http.host: "0.0.0.0"
log.level: info
path.logs: /data/logstash01/logs
xpack.monitoring.enabled: false
  1. Start the logstash command
/opt/logstash/bin/logstash -f /opt/logstash/config/logstash.yml
  1. Start the logstash service error log
Using bundled JDK: /opt/logstash/jdk
Sending Logstash logs to /data/logstash01/logs which is now configured via log4j2.properties
[2023-07-24T18:00:09,135][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/logstash/config/log4j2.properties
[2023-07-24T18:00:09,143][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.6.1", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +indy +jit [x86_64-linux]"}
[2023-07-24T18:00:09,145][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms10g, -Xmx10g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-07-24T18:00:09,298][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-07-24T18:00:09,893][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-07-24T18:00:10,053][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 19, column 1 (byte 322) after ", :backtrace=>["/opt/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:234:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:168:in `initialize'", "/opt/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/opt/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}
[2023-07-24T18:00:10,071][INFO ][logstash.runner          ] Logstash shut down.
[2023-07-24T18:00:10,076][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
	at opt.logstash.lib.bootstrap.environment.<main>(/opt/logstash/lib/bootstrap/environment.rb:91) ~[?:?]
  1. other
    There are attempts to configure the logstash system environment variables, custom configuration /opt/logstash/config/pipelines.yml
- pipeline.id: Egress_Firewall
  path.config: "/opt/logstash/config/Egress_Firewall.conf"
- pipeline.id: Linux_audit_log
  path.config: "/opt/logstash/config/Linux_audit_log.conf"

Logstash startup plus pipelines.yml path, startup failed!
/opt/logstash/bin/logstash does not have any parameters behind it, the config/(top) .conf file is read by default, and other .conf files cannot be read!

How to solve this problem? please!

2

This is your problem. If you use -f it should point to a configuration file like /opt/logstash/config/Egress_Firewall.conf, but if you want multiple pipelines then do not use the -f option at all, use pipelines.yml.

4

@Badger I have tried to configure pipelines.yml to define different pipelines, start without any parameters, but it can only process Egress_Firewall.conf normally, but not Linux_audit_log.conf, and the multi-pipelines defined in pipelines.yml does not take effect at all!

5

If you want to use pipelines.yml you need to remove this line from logstash.yml, your pipelines needs to be configured only in pipelines.yml and you need to start it as a service or without any parameters.

1 Like
6

@leandrojmp
Thanks for the help!

At first, the information I found was that pipelines.yml cannot be started with parameters, so the configuration file path was defined in logstash.yml. As a result, when logstash was started, the following warning message was always displayed:

[WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

The defined pipelines.yml did not work, and the startup failed!
After your reminder, try to start logstash with the following parameters:

/opt/logstash/bin/logstash --config.reload.automatic --path.settings /opt/logstash/config/

Finally, I saw that the multi-pipeline started successfully

[INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:Linux_audit_log, :Egress_Firewall], :non_running_pipelines=>[]}

I was misled by "defining pipelines.yml, logstash startup cannot take any parameters"! :joy:

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.