Logstash 7.1.1 not writing last_run in last_run_metadata_path

Hello guys! I'm experiencing a problem with logstash 7.1.1 running as a deamon on debian with multiple pipelines.

Despite the fact that record_last_run is true and last_run_metadata_path is specified, no file is saved.
The file is not even found in the home folder of root/logstash user.

I tried to change location and extend permissions but with no luck.
I was searching for this file because I would like to delete this file in order to force logstash to reload an index from scratch. I couldn't find it and therefore I deleted the index on elasticsearch. After a while I found that the entire index was populated again.
I wonder... If logstash is not tracking the last run, does this means that every minute it's going to load all the docs into ElasticSearch?

Any suggestion is welcome. Thank you in advance

It sounds like you are using a jdbc input. Can you show us the input configuration? (With sensitive data like passwords redacted.)

Ciao Badger, here a simplified version of the input block.

     input {
         jdbc {
             jdbc_driver_library => "/usr/share/java/mysql-connector-java-8.0.16.jar"
             jdbc_driver_class => "com.mysql.jdbc.Driver"
             jdbc_connection_string => "jdbc:mysql://servername:3306/databasename?useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=true&serverTimezone=UTC&zeroDateTimeBehavior=convertToNull"
             jdbc_user => "username"
             jdbc_password => "password"
             jdbc_paging_enabled => true
             schedule => "*/1 * * * *"
             statement => "SELECT field1,field2,UNIX_TIMESTAMP(timestamp_update) 
    as  tracking_field FROM table where timestamp_update < now() and UNIX_TIMESTAMP(timestamp_update)>:sql_last_value 
    order by timestamp_update asc"
                     use_column_value => true
                     tracking_column => tracking_field
                     tracking_column_type => "numeric"
                     record_last_run => true
                     last_run_metadata_path => "/storage/elk/logstash/run_ordsummary"

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.