I am trying to load data from Amazon Redshift, but it always exists Logstash process by throwing below exception.
Stacktrace
[2020-07-08T11:46:30,170][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:agent], :non_running_pipelines=>[]}
[2020-07-08T11:46:30,468][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>16060}
/mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated
[2020-07-08T11:48:00,394][ERROR][logstash.inputs.jdbc ][agent][b9b80c1d40de9a22b9af79023f0634a913716f23e6050d5b04c381d0a0be9fda] Unable to connect to database. Tried 1 times {:error_message=>"org.postgresql.Driver not loaded, try installing jdbc-postgres gem"}
{ 2054 rufus-scheduler intercepted an error:
2054 job:
2054 Rufus::Scheduler::CronJob "48 11 8 7 *" {}
2054 error:
2054 2054
2054 Sequel::AdapterNotFound
2054 org.postgresql.Driver not loaded, try installing jdbc-postgres gem
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/sequel-5.32.0/lib/sequel/adapters/jdbc.rb:56:in `load_driver'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/sequel-5.32.0/lib/sequel/adapters/jdbc/postgresql.rb:3:in `<main>'
2054 org/jruby/RubyKernel.java:974:in `require'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/sequel-5.32.0/lib/sequel/database/connecting.rb:88:in `load_adapter'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/sequel-5.32.0/lib/sequel/adapters/jdbc.rb:368:in `adapter_initialize'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/sequel-5.32.0/lib/sequel/database/misc.rb:154:in `initialize'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/sequel-5.32.0/lib/sequel/database/connecting.rb:57:in `connect'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/sequel-5.32.0/lib/sequel/core.rb:125:in `connect'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:119:in `block in jdbc_connect'
2054 org/jruby/RubyKernel.java:1442:in `loop'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:116:in `jdbc_connect'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:194:in `open_jdbc_connection'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:253:in `execute_statement'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/inputs/jdbc.rb:310:in `execute_query'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/inputs/jdbc.rb:277:in `block in run'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234:in `do_call'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258:in `do_trigger'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300:in `block in start_work_thread'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:299:in `block in start_work_thread'
2054 org/jruby/RubyKernel.java:1442:in `loop'
2054 /mnt/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289:in `block in start_work_thread'
2054 tz:
2054 ENV['TZ']:
2054 Time.now: 2020-07-08 11:48:00 +0000
2054 scheduler:
2054 object_id: 2052
2054 opts:
2054 {:max_work_threads=>1}
2054 frequency: 0.3
2054 scheduler_lock: #<Rufus::Scheduler::NullLock:0x52b8bb3>
2054 trigger_lock: #<Rufus::Scheduler::NullLock:0x88a9cd1>
2054 uptime: 89.611559 (1m29s613)
2054 down?: false
2054 threads: 2
2054 thread: #<Thread:0x483b54d4>
2054 thread_key: rufus_scheduler_2052
2054 work_threads: 1
2054 active: 1
2054 vacant: 0
2054 max_work_threads: 1
2054 mutexes: {}
2054 jobs: 1
2054 at_jobs: 0
2054 in_jobs: 0
2054 every_jobs: 0
2054 interval_jobs: 0
2054 cron_jobs: 1
2054 running_jobs: 1
2054 work_queue: 0
} 2054 .
^C
Configuration
input {
jdbc {
jdbc_driver_library => "./RedshiftJDBC42-1.2.10.1009.jar"
jdbc_driver_class => "com.amazon.redshift.jdbc42.Driver"
jdbc_connection_string => "jdbc:postgresql://xxx.us-west-2.redshift.amazonaws.com:5439/mdatabase"
jdbc_user => "abc"
jdbc_password => "abc"
jdbc_validate_connection => true
schedule => "17 11 8 7 *"
statement_filepath =>"queries/agent.sql"
}
}
output {
stdout { codec => dots }
elasticsearch {
hosts => "hostname:17002"
index => "agent_index_%{+YYYY_MM_dd}"
}
}
I always get below error when schedule is triggered for above configuration
Unable to connect to database. Tried 1 times {:error_message=>"org.postgresql.Driver not loaded, try installing jdbc-postgres gem"}
Note: Using ELK Version: 7.8.0, and OpenJDK version
openjdk version "1.8.0_192"
OpenJDK Runtime Environment (Zulu 8.33.0.2-linux64) (build 1.8.0_192-b01)
OpenJDK 64-Bit Server VM (Zulu 8.33.0.2-linux64) (build 25.192-b01, mixed mode)