# How to configure multi-pipeline configuration in logstash 8.6.1? **URL:** \ **Category:** Logstash\ **Created:** [July 24, 2023, 10:22am UTC](https://discuss.elastic.co/t/how-to-configure-multi-pipeline-configuration-in-logstash-8-6-1/339071 "2023-07-24T10:22:08Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Koele](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@Koele](https://discuss.elastic.co/u/Koele)\ **Post date:** [July 24, 2023, 10:22am UTC](https://discuss.elastic.co/t/how-to-configure-multi-pipeline-configuration-in-logstash-8-6-1/339071/1 "2023-07-24T10:22:08Z")
1. Basic Information ```auto logstash version: 8.6.1 Logstash installation method: tar.gz logstash installation directory: /opt OS: CentOS 7 ``` 1. ogstash.yml configuration file ```auto node.name: logstash01 path.data: /data/logstash01 pipeline.ordered: auto path.config: /opt/logstash/config/*.conf http.host: "0.0.0.0" log.level: info path.logs: /data/logstash01/logs xpack.monitoring.enabled: false ``` 1. Start the logstash command ```auto /opt/logstash/bin/logstash -f /opt/logstash/config/logstash.yml ``` 1. Start the logstash service error log ```auto Using bundled JDK: /opt/logstash/jdk Sending Logstash logs to /data/logstash01/logs which is now configured via log4j2.properties [2023-07-24T18:00:09,135][INFO][logstash.runner] Log4j configuration path used is: /opt/logstash/config/log4j2.properties [2023-07-24T18:00:09,143][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.6.1", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +indy +jit [x86_64-linux]"} [2023-07-24T18:00:09,145][INFO][logstash.runner] JVM bootstrap flags: [-Xms10g, -Xmx10g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED] [2023-07-24T18:00:09,298][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2023-07-24T18:00:09,893][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [2023-07-24T18:00:10,053][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 19, column 1 (byte 322) after ", :backtrace=>["/opt/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:234:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:168:in `initialize'", "/opt/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/opt/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]} [2023-07-24T18:00:10,071][INFO][logstash.runner] Logstash shut down. [2023-07-24T18:00:10,076][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?] at opt.logstash.lib.bootstrap.environment.
(/opt/logstash/lib/bootstrap/environment.rb:91) ~[?:?] ``` 1. other There are attempts to configure the logstash system environment variables, custom configuration /opt/logstash/config/pipelines.yml ```auto - pipeline.id: Egress_Firewall path.config: "/opt/logstash/config/Egress_Firewall.conf" - pipeline.id: Linux_audit_log path.config: "/opt/logstash/config/Linux_audit_log.conf" ``` Logstash startup plus pipelines.yml path, startup failed! /opt/logstash/bin/logstash does not have any parameters behind it, the config/(top) .conf file is read by default, and other .conf files cannot be read! How to solve this problem? please! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 24, 2023, 4:44pm UTC](https://discuss.elastic.co/t/how-to-configure-multi-pipeline-configuration-in-logstash-8-6-1/339071/2 "2023-07-24T16:44:14Z")
> [@Koele](#): > > `-f /opt/logstash/config/logstash.yml` This is your problem. If you use -f it should point to a configuration file like /opt/logstash/config/Egress\_Firewall.conf, but if you want multiple pipelines then do not use the -f option at all, use pipelines.yml. ---
**Author:** ![Koele](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@Koele](https://discuss.elastic.co/u/Koele)\ **Post date:** [July 25, 2023, 1:20am UTC](https://discuss.elastic.co/t/how-to-configure-multi-pipeline-configuration-in-logstash-8-6-1/339071/4 "2023-07-25T01:20:11Z")
@Badger I have tried to configure pipelines.yml to define different pipelines, start without any parameters, but it can only process Egress\_Firewall.conf normally, but not Linux\_audit\_log.conf, and the multi-pipelines defined in pipelines.yml does not take effect at all! ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 25, 2023, 2:38am UTC](https://discuss.elastic.co/t/how-to-configure-multi-pipeline-configuration-in-logstash-8-6-1/339071/5 "2023-07-25T02:38:26Z")
> [@Koele](#): > > `path.config: /opt/logstash/config/*.conf` If you want to use `pipelines.yml` you need to remove this line from `logstash.yml`, your pipelines needs to be configured only in `pipelines.yml` and you need to start it as a service or without any parameters. ---
**Author:** ![Koele](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@Koele](https://discuss.elastic.co/u/Koele)\ **Post date:** [July 25, 2023, 3:26am UTC](https://discuss.elastic.co/t/how-to-configure-multi-pipeline-configuration-in-logstash-8-6-1/339071/6 "2023-07-25T03:26:59Z")
@leandrojmp Thanks for the help! At first, the information I found was that pipelines.yml cannot be started with parameters, so the configuration file path was defined in logstash.yml. As a result, when logstash was started, the following warning message was always displayed: ```auto [WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified ``` The defined pipelines.yml did not work, and the startup failed! After your reminder, try to start logstash with the following parameters: ```auto /opt/logstash/bin/logstash --config.reload.automatic --path.settings /opt/logstash/config/ ``` Finally, I saw that the multi-pipeline started successfully ```auto [INFO][logstash.agent] Pipelines running {:count=>2, :running_pipelines=>[:Linux_audit_log, :Egress_Firewall], :non_running_pipelines=>[]} ``` I was misled by " **defining pipelines.yml, logstash startup cannot take any parameters**"! 😂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 22, 2023, 3:27am UTC](https://discuss.elastic.co/t/how-to-configure-multi-pipeline-configuration-in-logstash-8-6-1/339071/7 "2023-08-22T03:27:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.