LogStash reading data from Oracle database using Oracle Wallet

Elastic Stack Logstash
1

Our security team has strict policy using the password in the configuration file so we have to use wallet. I am trying to setup Logstah to work with Oracle wallet. has anyone successfully configured with Oracle wallet? I don't have much experience with ruby so bear with me if it is silly question. I understand Logstash uses ruby libraries underneath and ruby runs on JDK . We never had problem using wallet from Java. As per my knowledge, if java supports it, ruby should unless if its not invoking the right API's. If jdbc ruby does not support wallet, is there way to configure password in separate file and pass that file path as value for "jdbc_password" in Logstash configuration?

Here is the Logstash configuration

jdbc {
jdbc_connection_string => "jdbc:oracle:thin:/@WALLET"
jdbc_user => ""
jdbc_password => ""
jdbc_driver_library => "/jdbc/lib/ojdbc6.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_fetch_size => 50
statement => "SELECT * from MYTABLE partition(P20160520)"
type => "db-mytable"
use_column_value => true
tracking_column => "MSG_ID"
#last_run_metadata_path => "../logstash_jdbc_last_run"
record_last_run => true
clean_run => false
}

Exception

Pipeline aborted due to error {:exception=>#<Sequel::DatabaseConnectionError: Java::JavaSql::SQLRecoverableException:## IO Error: Unknown host specified>, :backtrace=>["oracle.jdbc.driver.T4CConnection.logon(oracle/jdbc/driver/T4CConnection.java:419)", "oracle.jdbc.driver.PhysicalConnection.(oracle/jdbc/driver/PhysicalConnection.java:536)", "oracle.jdbc.driver.T4CConnection.(oracle/jdbc/driver/T4CConnection.java:228)", "oracle.jdbc.driver.T4CDriverExtension.getConnection(oracle/jdbc/driver/T4CDriverExtension.java:32)", "oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:521)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:606)", "RUBY.connect(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/adapters/jdbc.rb:222)", "RUBY.make_new(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool.rb:103)", "RUBY.make_new(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:224)", "RUBY.available(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:197)", "RUBY._acquire(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:133)", "RUBY.acquire(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:147)", "RUBY.sync(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:265)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:149)", "RUBY.sync(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:265)", "RUBY.acquire(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:146)", "RUBY.hold(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/connection_pool/threaded.rb:104)", "RUBY.synchronize(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/database/connecting.rb:256)", "RUBY.test_connection(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/sequel-4.33.0/lib/sequel/database/connecting.rb:266)", "RUBY.prepare_jdbc_connection(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-input-jdbc-3.0.2/lib/logstash/plugin_mixins/jdbc.rb:170)", "RUBY.register(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-input-jdbc-3.0.2/lib/logstash/inputs/jdbc.rb:167)", "RUBY.start_inputs(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:330)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.start_inputs(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:329)", "RUBY.start_workers(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:180)", "RUBY.run(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:136)", "RUBY.start_pipeline(/home/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb:465)", "java.lang.Thread.run(java/lang/Thread.java:745)"], :level=>:error, :file=>"logstash/agent.rb", :line=>"467", :method=>"start_pipeline"}
stopping pipeline {:id=>"main", :file=>"logstash/agent.rb", :line=>"404", :method=>"shutdown_pipelines"}

2

Pipeline aborted due to error {:exception=>#IO Error: Unknown host specified>, :backtrace=>["oracle.jdbc.driver.T4CConnection.logon(oracle/jdbc/driver/T4CConnection.java:419)",

To me this indicates a problem with the connection string.

If jdbc ruby does not support wallet, is there way to configure password in separate file and pass that file path as value for "jdbc_password" in Logstash configuration?

Yes, that functionality was added a few weeks ago (PR #128) and included in logstash-input-jdbc 4.0.2.

3

when using wallet, connection string passes the alias name which matches service name in tnsnames.ora. The connection string will not contain host name and port number

jdbc_connection_string => "jdbc:oracle:thin:/@WALLET"

You can find more explanation here

I looked at jdbc.rb code, i think it need to be enhanced to support Oracle Wallet .I debugged code little bit. I printed the props and args to console. The code is passing the userid , password which is not required in case of wallet. I think "driver" class might be parsing the connection string and passing the host name and port number. where can i report this issue?

driver.new.connect(args[0], props)

4

Issues with the jdbc input plugin can be reported here: https://github.com/logstash-plugins/logstash-input-jdbc/issues/new

5

Thanks Magnus. I filed issue https://github.com/logstash-plugins/logstash-input-jdbc/issues/135