Multiple Pipelines throwing errors when enabled

Good Afternoon Elastic Peeps;

I have been working with configuring and employing multiple pipelines in my ELK stack. I have implemented ELK 7.13.4 and when I use a simple syslog pipeline I am able to gather the information from Cisco Routers, switches etc. I am now working on a second pipeline that is meant to ingest Microsoft SQL database information. When I enable the second pipeline, everything comes to a stop. I do not receive any syslogs into my Logstash. I also get the following error in the /var/log/logstash/logstash-plian.log

[2021-12-14T13:35:54,456][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>ArgumentError, :message=>"Only pipeline related settings are expected. Received \"pipeline-id\". Allowed settings: [\"config.debug\", \"config.support_escapes\", \"config.reload.automatic\", \"config.reload.interval\", \"config.string\", \"dead_letter_queue.enable\", \"dead_letter_queue.flush_interval\", \"dead_letter_queue.max_bytes\", \"metric.collect\", \"pipeline.java_execution\", \"pipeline.plugin_classloaders\", \"path.config\", \"path.dead_letter_queue\", \"path.queue\", \"pipeline.batch.delay\", \"pipeline.batch.size\", \"pipeline.id\", \"pipeline.reloadable\", \"pipeline.system\", \"pipeline.workers\", \"pipeline.ordered\", \"pipeline.ecs_compatibility\", \"queue.checkpoint.acks\", \"queue.checkpoint.interval\", \"queue.checkpoint.writes\", \"queue.checkpoint.retry\", \"queue.drain\", \"queue.max_bytes\", \"queue.max_events\", \"queue.page_capacity\", \"queue.type\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:147:in `block in merge_pipeline_settings'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:145:in `merge_pipeline_settings'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/multi_local.rb:36:in `block in pipeline_configs'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/multi_local.rb:34:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:76:in `block in fetch'", "org/jruby/RubyArray.java:2572:in `collect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:75:in `fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:188:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:126:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:409:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[2021-12-14T13:35:54,463][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Only pipeline related settings are expected. Received \"pipeline-id\". Allowed settings: [\"config.debug\", \"config.support_escapes\", \"config.reload.automatic\", \"config.reload.interval\", \"config.string\", \"dead_letter_queue.enable\", \"dead_letter_queue.flush_interval\", \"dead_letter_queue.max_bytes\", \"metric.collect\", \"pipeline.java_execution\", \"pipeline.plugin_classloaders\", \"path.config\", \"path.dead_letter_queue\", \"path.queue\", \"pipeline.batch.delay\", \"pipeline.batch.size\", \"pipeline.id\", \"pipeline.reloadable\", \"pipeline.system\", \"pipeline.workers\", \"pipeline.ordered\", \"pipeline.ecs_compatibility\", \"queue.checkpoint.acks\", \"queue.checkpoint.interval\", \"queue.checkpoint.writes\", \"queue.checkpoint.retry\", \"queue.drain\", \"queue.max_bytes\", \"queue.max_events\", \"queue.page_capacity\", \"queue.type\"]"}
[2021-12-14T13:35:54,539][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-12-14T13:35:59,573][INFO ][logstash.runner          ] Logstash shut down.
[2021-12-14T13:35:59,579][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

If there is anyone that can help shed some light on what I am missing that would be fantastic. Thank you.

That suggests you have pipeline-id where you should have pipeline.id.

Thank you Badger. That got me one step closer. I needed those fresh set of eyes. It is now throwing this error:

[2021-12-14T13:57:43,572][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:gac, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ArgumentError) URI is not valid - host is not specified", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}

OK, so now you are back to this issue.

Yes, Yes I am am. :frowning: Not sure what or where to go from here. I imported the sqljdbc_6.0.8112.200_enu.tar.gz file and extracted it to /etc/logstash/sqljdbc6.0. In the configuration file I am creating I pointed it to the location. Here is a cleaned up version of the jdbc.conf file.

input {
  jdbc {
    jdbc_driver_library => "/etc/logstash/sqljdbc_6.0/enu/jre8/sqljdbc42.jar"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://SQL_Db_Server;databaseName=McAfeeEPO"
    jdbc_user => "SQL user"
    jdbc_password => "SQL_Pass"
    statement => "SELECT

As I said in the other thread, logstash is trying to validate a plugin option that is a :uri. None of the jdbc input options are :uri, so that is not the filter you should be looking at.

Ok Thank you Badger.

From what i posted above it doesn't look like a uri is being called for. I will continue looking through my configuration to see why it would be looking for a uri.

Do you mean that it is looking at an IP address and needs a fqdn? I am testing that theory at the moment.

No, that's not what I meant. The code calls java.net.URI.new and the exception tells you that the host component in the resulting object is nil. It is inconceivable to me that java.net.URI does not support IP addresses.

OK thank you. I am verifying with the Linux guru on my team that we actually have java installed on the logstasth server. If it is not then that could be what is blocking us.

Thank you.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.