Logstash stops treatment when database goes down

Hi, I have multiple JDBC inputs waiting for PostgreSQL database lines each minute. The database has a problem and is down. Due to this mistake, I don't receive any log in Kibana.

I tried to find some fields in the documentation to avoid blocking the stack but except "jdbc_validate_connection", which doesn't solve the problem, I don't have any other idea.

Here is the error I got:

[2023-07-19T15:00:34,960][ERROR][logstash.inputs.jdbc     ][main] Unable to connect to database. Tried 1 times {:message=>"Java::OrgPostgresqlUtil::PSQLException: The connection attempt failed.", :exception=>Sequel::DatabaseConnectionError, :cause=>#<Java::OrgPostgresqlUtil::PSQLException: The connection attempt failed.>, :backtrace=>["org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(org/postgresql/core/v3/ConnectionFactoryImpl.java:354)", "org.postgresql.core.ConnectionFactory.openConnection(org/postgresql/core/ConnectionFactory.java:54)", "org.postgresql.jdbc.PgConnection.<init>(org/postgresql/jdbc/PgConnection.java:263)", "org.postgresql.Driver.makeConnection(org/postgresql/Driver.java:443)", "org.postgresql.Driver.connect(org/postgresql/Driver.java:297)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:472)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:330)", "RUBY.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.65.0/lib/sequel/adapters/jdbc.rb:237)", "RUBY.new_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.65.0/lib/sequel/database/connecting.rb:248)", "RUBY.make_new(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.65.0/lib/sequel/connection_pool.rb:154)", "RUBY.assign_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.65.0/lib/sequel/connection_pool/threaded.rb:209)", "RUBY.acquire(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.65.0/lib/sequel/connection_pool/threaded.rb:139)", "RUBY.hold(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.65.0/lib/sequel/connection_pool/threaded.rb:91)", "RUBY.synchronize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.65.0/lib/sequel/database/connecting.rb:293)", "RUBY.test_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.65.0/lib/sequel/database/connecting.rb:302)", "RUBY.initialize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.65.0/lib/sequel/database/misc.rb:188)", "org.jruby.RubyClass.new(org/jruby/RubyClass.java:890)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "RUBY.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.65.0/lib/sequel/database/connecting.rb:57)", "RUBY.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.65.0/lib/sequel/core.rb:124)", "RUBY.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:118)", "org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1507)", "org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen)", "RUBY.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:115)", "RUBY.open_jdbc_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:158)", "RUBY.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:309)", "RUBY.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48)", "RUBY.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "RUBY.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236)", "RUBY.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:395)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:320)", "RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194)", "RUBY.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:309)", "java.lang.Thread.run(java/lang/Thread.java:833)"]}
[2023-07-19T15:00:35,159][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Can't create a connection pool to the database>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:318:in `register'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:395:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:320:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x7b87fb72@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-07-19T15:00:35,169][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

I looked for a field which verifies if the connection can be established: if yes, get entries, else, wait the next attempt. I'll continue my research but it would be great if someone is able to help me :slight_smile:
Thanks in advance!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.