Logstash not updating last run metadata file

Elastic Stack Logstash
1

In my Logstash I want to download from a database the most recent data using :sql_last_value in a query and tracking_column option in conf file. I've set last_run_metadata_path because I have 2 pipelines for the same table but Logstash saved last date only once or stopped saving new dates and now I can see in logs that it runs queries with the same :sql_last_value from metadata file.

That's how my conf file looks like, it has many jdbc inputs and one of them below:

  jdbc {
        jdbc_driver_library => "/opt/logstash/lib/ojdbc8.jar"
        jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
        jdbc_connection_string => ""
        jdbc_user => ""
        jdbc_password => ""
        schedule => "*/15 * * * *"
        statement_filepath => "/etc/logstash/queries/UAT/transactions_UAT.sql"
        use_column_value => true
        tracking_column => 'sys_created_on'
        tracking_column_type => "timestamp"
        last_run_metadata_path => "/etc/logstash/conf.d/lastrun_metadata/transactions_uat_metadata"
        tags => ["transactions_uat"]
  }

Content of the metadata file:

--- 2018-05-26 08:41:55.000000000 -04:00
I can see in the logs that Logstash always uses the same date from the metadata file and newer updates it:

select * from snc_uat.syslog_transaction0007
 where "sys_created_on" >= TIMESTAMP '2018-05-26 08:41:55.000000 -04:00'

Logstash is working and is downloading recent data but unnecessarily processes data that already exists. Why is Logstash not updating metadata?

1 Like
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.