How to re-index a jdbc entity based on timestamp

Elastic Stack Logstash
1

Hi,

I'm trying to do an incremental update of the index after every scheduled index.
How to generate a metadata file which will contain the lastexecutiondate (timestamp) and then re-use that value for incremental update.

input {
jdbc {
last_run_metadata_path => "meta.json"
jdbc_connection_string => "jdbc:oracle:thin:/@::"
jdbc_user => "user"
jdbc_password => "passwd"
jdbc_validate_connection => true
jdbc_driver_library => "/scratch/Logstash/ojdbc8.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"

This is where i'm confused

statement => "SELECT ID from projects where last_update_date > ?"
parameters => {"timestamp" => "$metrics.lastexecutionstart"}
}
}
output {
elasticsearch {
index => "cdrm"
document_type => "projects"
document_id => "%{project_id}"
hosts => ["localhost:9200"]
}
stdout { codec => json }
}

Thanks
Thomas

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.