Last_run_metadata_path not updated with sql_last_value

Elastic Stack Logstash
1

Hi, I m new to ELK and I'm having an issue with the jdbc sql_last_value. So I'm trying to update my Elastic database from a SQL database using the jdbc driver, I want to update records that are modified (dhm is the row that contains the last modification date). The SQL query read the sql_last_value contained in last_run_metadata_path (when I write manualy a sql_last_value in the file it works), but it's not updated with a new one as it should (if I have well understood how it works at least)
Here is my config file :

input {
    jdbc {
        type => "client"
        jdbc_driver_library => "${LOGSTASH_JDBC_DRIVER_JAR_LOCATION}"
        jdbc_driver_class => "${LOGSTASH_JDBC_DRIVER}"
        jdbc_connection_string => "${LOGSTASH_JDBC_URL}"
        jdbc_user => "${LOGSTASH_JDBC_USERNAME}"
        jdbc_password => "${LOGSTASH_JDBC_PASSWORD}"
        jdbc_paging_enabled => true
        jdbc_page_size => 50
		#jdbc_fetch_size => 100
		#check internet
        jdbc_default_timezone => "UTC"
        # exécuté toutes les 20 sec
        schedule => "*/20 * * * * *"
		statement_filepath => "/usr/share/logstash/config/queries/req_logstash_CLIENT.sql"
        use_column_value => true
        tracking_column => "dhm"
        tracking_column_type => "timestamp"
        record_last_run => true
        clean_run => false
        tags => ["lastupdated_client"]
        last_run_metadata_path => "/usr/share/logstash/temporary/logstash-jdbc-last-run-info_client.yml"

    }
    jdbc {
        type => "commande"
        jdbc_driver_library => "${LOGSTASH_JDBC_DRIVER_JAR_LOCATION}"
        jdbc_driver_class => "${LOGSTASH_JDBC_DRIVER}"
        jdbc_connection_string => "${LOGSTASH_JDBC_URL}"
        jdbc_user => "${LOGSTASH_JDBC_USERNAME}"
        jdbc_password => "${LOGSTASH_JDBC_PASSWORD}"
        jdbc_paging_enabled => true
        jdbc_page_size => 50
		#jdbc_fetch_size => 100
        jdbc_default_timezone => "UTC"
        # exécuté toutes les 20 sec
        schedule => "*/20 * * * * *"
		statement_filepath => "/usr/share/logstash/config/queries/req_logstash_COMMANDE.sql"
		use_column_value => true
        tracking_column => "dhm"
        tracking_column_type => "timestamp"
		record_last_run => true
        clean_run => false
        tags => ["lastupdated_commande"]
        last_run_metadata_path => "/usr/share/logstash/temporary/logstash-jdbc-last-run-info_commande.yml"
    }
    
}

If you have any idea about why the logstash-jdbc-last-run-info_commande.yml (the other input isn't being updated neither) isn't being upadated with the sql_last_value please consider answering me.

Thanks for your time :slight_smile:

2

Are you sure the user running logstash has permission to update that file?

3

Thank you for your answer,
Yes I checked it I have read and write permissions
What's weird is that with another config and another database (SQL postgre database) it works....
I have the same config with just other filters and outputs
And when I manualy change the sql_last_value during the process it doesn't update in the querry...

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.