Do not update sql_last_value when an Elastic error is encountered

Elastic Stack Logstash
1 
input {
    jdbc {
        jdbc_driver_library => "/usr/share/logstash/drivers/jdbc/${PG_DRIVER}"
        jdbc_driver_class => "org.postgresql.Driver"
        jdbc_connection_string => "jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_NAME}?user=${DATABASE_USER}"
        jdbc_user => "${DATABASE_USER}"
        jdbc_password => "${DATABASE_PASSWORD}"
        schedule => "* * * * *"
        statement_filepath => "/usr/share/logstash/inputs/query.sql"
        clean_run => true
        last_run_metadata_path => "/usr/share/logstash/.logstash_jdbc_last_run"
    }
}

My logstash.cgf file uses the above input to query a postgres database and send the results to Elastic. The query itself uses the sql_last_value to only ingest new data. However, if logstash loses the connection to elastic the sql_last_value continues to update resulting in lost data when the elastic connection is reestablished.

Is there anyway to keep sql_last_value from updating when an elastic error is encountered?

Thank you in advance,
K Smith

2

No. The input updates sql_last_value when the query is complete, it has no way to know what happens at the other end of the pipeline.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.