LS failing to start with JDBC or multiple Input plugins

Elastic Stack Logstash
1

Hi
My LS config has 2 input plugins 1-FileBeats & 2-JDBC. Here are my queries or issues.

  1. LS is failing to start with JDBC plugin and the database listner is down, even though the jdbc_validate_connection is set to false and the schedule is in the future. Ideally if the jdbc_validate_connection is false then the LS should try to open the connection at scheduled execution.
    The target database is a mirror database and gets down every night to sync up with master production DB.

  2. The LS is able to start if I disable JDBC plugin and start with only FB plugin. As of now I have 2 FB clients to LS.

LS Startup Errors:
**1-When only JDBC Plugin is set **
Settings: Default filter workers: 2
The error reported is:
Java::JavaSql::SQLException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

**2-When both Filebeat & JDBC Plugin is set **
Settings: Default filter workers: 2
The error reported is:
Java::JavaSql::SQLException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor
Beats input: unhandled exception {:exception=>#<ConcurrencyError: interrupted waiting for mutex: null>, :backtrace=>["org/jruby/ext/thread/Mutex.java:94:in lock'", "org/jruby/ext/thread/Mutex.java:147:insynchronize'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-lumberjack-2.0.4/lib/logstash/circuit_breaker.rb:89:in state'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-lumberjack-2.0.4/lib/logstash/circuit_breaker.rb:36:inexecute'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/logstash/inputs/beats.rb:111:in run'", "org/jruby/RubyProc.java:271:incall'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/logstash/inputs/beats.rb:150:in create_event'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-codec-plain-2.0.2/lib/logstash/codecs/plain.rb:35:indecode'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-codec-multiline-2.0.4/lib/logstash/codecs/identity_map_codec.rb:143:in decode'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/logstash/inputs/beats.rb:144:increate_event'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/logstash/inputs/beats.rb:185:in invoke'", "org/jruby/RubyProc.java:271:incall'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:382:in data'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:361:inread_socket'", "org/jruby/RubyProc.java:271:in call'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:373:inack_if_needed'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:357:in read_socket'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:246:injson_data_payload'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:163:in feed'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:296:incompressed_payload'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:163:in feed'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:342:inread_socket'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:319:in run'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/logstash/inputs/beats.rb:184:ininvoke'", "org/jruby/RubyProc.java:271:in call'", "/sysapp/app/elk/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/concurrent-ruby-0.9.2-java/lib/concurrent/executor/executor_service.rb:515:inrun'", "Concurrent$$JavaExecutorService$$Job_84093389.gen:13:in `run'"], :level=>:error}
A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Beats port=>5044, codec=><LogStash::Codecs::Plain charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, congestion_threshold=>5, target_field_for_codec=>"message">
Error: Concurrent::RejectedExecutionError {:level=>:error}

2

This is an Oracle-specific error. Googling the error message results in lots of hits—have you looked at them? For example, the first hit I get indicates that it could be a connection string syntax problem.

3

My Query was - Does the LS establish the connection to database at the start up itself even though the JDBC schedule is set to future date & time and the jdbc_validate_connection is false !!

It seems LS is trying to test the connection at the startup itself. It would be nice if LS doesn't test the connection at the startup if jdbc_validate_connection is false...

4

That parameter seems to be connected to the connection pool configuration. Logstash will unconditionally attempt to connect to the database when the plugin is started. Note the unconditional call to jdbc_connect() from the unconditionally called prepare_jdbc_connection().