I am experiencing an issue when trying to use the tracking_column_type="timestamp" and use_column_value=true properties with the jdbc input plugin.
I keep getting the below error:
{ 2012 rufus-scheduler intercepted an error:
2012 job:
2012 Rufus::Scheduler::CronJob "* * * * *" {}
2012 error:
2012 2012
2012 TypeError
2012 can't dup NilClass
2012 org/jruby/RubyKernel.java:1882:in dup' 2012 uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/date/format.rb:838:in_parse'
2012 uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/date.rb:1830:in parse' 2012 /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/value_tracking.rb:87:inset_value'
2012 /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/plugin_mixins/jdbc.rb:237:in execute_statement' 2012 /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb:264:inexecute_query'
2012 /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.9/lib/logstash/inputs/jdbc.rb:245:in block in run' 2012 /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234:indo_call'
2012 /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258:in do_trigger' 2012 /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300:inblock in start_work_thread'
2012 /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:299:in block in start_work_thread' 2012 org/jruby/RubyKernel.java:1292:inloop'
2012 /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289:in `block in start_work_thread'
2012 tz:
2012 ENV['TZ']:
2012 Time.now: 2018-05-22 12:28:19 UTC
2012 scheduler:
2012 object_id: 2008
2012 opts:
2012 {:max_work_threads=>1}
2012 frequency: 0.3
2012 scheduler_lock: #Rufus::Scheduler::NullLock:0x606c9a04
2012 trigger_lock: #Rufus::Scheduler::NullLock:0x4577720d
2012 uptime: 122.325062 (2m2s325)
2012 down?: false
2012 threads: 2
2012 thread: #Thread:0x2997006d
2012 thread_key: rufus_scheduler_2008
2012 work_threads: 1
2012 active: 1
2012 vacant: 0
2012 max_work_threads: 1
2012 mutexes: {}
2012 jobs: 1
2012 at_jobs: 0
2012 in_jobs: 0
2012 every_jobs: 0
2012 interval_jobs: 0
2012 cron_jobs: 1
2012 running_jobs: 1
2012 work_queue: 0
} 2012 .
My configuration is as follows:
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/db"
# The user we wish to execute our statement as
jdbc_user => "root"
jdbc_password => ""
# The path to our downloaded jdbc driver
jdbc_driver_library => "/usr/share/logstash/config/mysql-connector-java-8.0.11.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_fetch_size => "10000"
schedule => "* * * * *"
# our query
statement => "SELECT u.id, u.FirstName as first_name, u.LastName as last_name, TIMESTAMPDIFF(YEAR, up.DateOfBirth, CURDATE()) AS age, up.Gender, up.Ethnicity, up.Language, up.Religion, up.ProfileName as profile_name, up.HasPhoto as has_photo, up.AdminPhotoRating as admin_rating, um.LastActivityAt as last_activity_date, um.LastLoginAt as last_login_date, l.Latitude, l.Longitude, l.Name AS location_name, u.ModifiedAt FROM users u INNER JOIN userprofiles up ON u.id = up.UserId INNER JOIN usermetadata um ON um.UserId = u.Id INNER JOIN locations l ON up.LocationId = l.id WHERE u.ModifiedAt > :sql_last_value ORDER BY u.ModifiedAt"
use_column_value => true
tracking_column => "ModifiedAt"
tracking_column_type => "timestamp"
last_run_metadata_path => "/usr/share/logstash/config/.logstash_jdbc_last_run"
}
}
output {
stdout { codec => json_lines }
elasticsearch {
hosts => ["localhost:9200"]
index => "test-migrate"
document_type => "data"
document_id => "%{id}"
user => "elastic"
password => ""
}
}
I can verify that the data returned from the SQL statement, especially the ModifiedAt column, does not contain any null values.
If I change the jdbc use_column_value property to false, then I get no errors, but this is not my desired result, as I need the value from the ModifiedAt column.
Is there something obvious that I am missing here?
Thanks in advance.