Jdbc: record_last_run always saves 2017-01-01 as last run date

[sivabalan19]

Hi ,

I am using Logstash 5.4.0. Trying to pull all the new records that are in my Db after my last run. Below is my configuration. However, I am finding the content of the file mentioned in last_run_metadata_path always revert back to "--- 2017-01-01 00:00:00.000000000 Z" after each run. It looks like it is not storing the time when schedular has run last time. Just to add, my tracking_column's type is timestamp and i have mentioned the same in configuration sql statement also (but for now i have masked it).
Because of this, i m getting duplicates from output. Help me please.

input {
jdbc {
type => "Database"
sql_log_level => "debug"
jdbc_driver_library => "/home/siva/Oracle/engine/logstash-5.4.0/vendor/ojdbc7.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:@localhost:1521/XE"
jdbc_user => "XXXX"
jdbc_password => "XXXX"
statement => "*****"
use_column_value => true
tracking_column => "insert_time"
tracking_column_type => "timestamp"
schedule => "* * * * *"
record_last_run => true
last_run_metadata_path => "/home/siva/Oracle/engine/logstash-5.4.0/bin/logstash_jdbc_last_run"
}
}

output {
stdout {
codec => json
}
}

[sivabalan19]

Can someone please help me here .

[guyboertje]

When you set use_column_value => true then sql_last_value becomes the value of the tracking column from each record as it is processed and the last record wins (and is persisted). In this case make sure that the query has an ORDER BY insert_time clause. Make sure that you have a field called insert_time in your SELECT clause. Surely not all insert_time values are "2017-01-01 00:00:00.000000000 Z"?.

When you set it to false then the sql_last_value becomes a timestamp taken at the time just before the query is run and that value is persisted at the end of the run.

[sivabalan19]

@guyboertje Thanks for that. It helped. I am able to find the last run date-time in the file now.
Appreciate your help.

-thanks,
Siva V