Logstash using multiple configuration

Elastic Stack Logstash
1

Hello again,

From the previous threads that I have solved, now I have two Logstash configurations, one that output to Http pluging and one to elasticsearch plugin.

I am configuring my D:\logstash-7.17.28\config\pipelines.yml this way:

 - pipeline.id: test
   pipeline.workers: 1
   pipeline.batch.size: 1
   queue.type: persisted
   queue.page_capacity: 64mb
   queue.max_bytes: 1024mb
   path.config: "D:/logstash-7.17.28/elasticsearch-logstash-v2.conf"
 - pipeline.id: another_test
   pipeline.workers: 1
   pipeline.batch.size: 1
   queue.type: persisted
   queue.page_capacity: 64mb
   queue.max_bytes: 1024mb
   path.config: "D:/logstash-7.17.28/http-logstash.conf"

but I receive an error message saying:

D:\logstash-7.17.28\bin>logstash.bat
"Using bundled JDK: D:\logstash-7.17.28\jdk\bin\java.exe"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to D:/logstash-7.17.28/logs which is now configured via log4j2.properties
[2025-03-17T16:28:11,872][INFO ][logstash.runner          ] Log4j configuration path used is: D:\logstash-7.17.28\config\log4j2.properties
[2025-03-17T16:28:11,883][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.28", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.26+4 on 11.0.26+4 +indy +jit [mswin32-x86_64]"}
[2025-03-17T16:28:11,885][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2025-03-17T16:28:16,699][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2025-03-17T16:28:16,704][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2025-03-17T16:28:21,999][INFO ][logstash.runner          ] Logstash shut down.
[2025-03-17T16:28:22,009][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at D_3a_.logstash_minus_7_dot_17_dot_28.lib.bootstrap.environment.<main>(D:\logstash-7.17.28\lib\bootstrap\environment.rb:94) ~[?:?]

What am I doing wrong? Why do I receive a message saying: No configuration found in the configured sources?

2

Just to complete the question, if I force the use of the D:\logstash-7.17.28\config\pipelines.yml I receive another error:

D:\logstash-7.17.28\bin>logstash.bat -f D:\logstash-7.17.28\config\pipelines.yml
"Using bundled JDK: D:\logstash-7.17.28\jdk\bin\java.exe"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to D:/logstash-7.17.28/logs which is now configured via log4j2.properties
[2025-03-17T16:50:25,719][INFO ][logstash.runner          ] Log4j configuration path used is: D:\logstash-7.17.28\config\log4j2.properties
[2025-03-17T16:50:25,730][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.28", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.26+4 on 11.0.26+4 +indy +jit [mswin32-x86_64]"}
[2025-03-17T16:50:25,733][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2025-03-17T16:50:25,821][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2025-03-17T16:50:32,010][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2025-03-17T16:50:32,603][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 9, column 2 (byte 333) after ", :backtrace=>["D:/logstash-7.17.28/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "D:/logstash-7.17.28/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "D:/logstash-7.17.28/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "D:/logstash-7.17.28/logstash-core/lib/logstash/agent.rb:392:in `block in converge_state'"]}
[2025-03-17T16:50:37,822][INFO ][logstash.runner          ] Logstash shut down.
[2025-03-17T16:50:37,834][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at D_3a_.logstash_minus_7_dot_17_dot_28.lib.bootstrap.environment.<main>(D:\logstash-7.17.28\lib\bootstrap\environment.rb:94) ~[?:?]
3

Are the files in the correct path?

This issue normally happens when the files cannot be found.

4

You should start LS:

D:\logstash-7.17.28\bin>logstash.bat --path.settings D:\logstash-7.17.28\config

In pipeline.yml should change to the backslash, without quotes:
path.config: D:\logstash-7.17.28\elasticsearch-logstash-v2.conf
or this notation for quotes.
path.config: "D:\\logstash-7.17.28\\http-logstash.conf"

5

Thanks everyone. Yes, the path was wrong.