sql_last_value update with Paging
-
I am configuring logstash to use jdbc input knowing that I am using jdbc_paging with the configuration.
-
what I am facing now is that sql_last_value is being updated after all records in the db is completed which is not what I want, I need to have sql_last_value being updated between paging such that in case logstash pod is down for any reason, it will be able to continue from the sql_last_value for which the last paging query finished successfully.
-
Is this something possible?
-
when I am not using the paging, and only default jdbc input, it is able to do that and I can see the last_metadata_path_file is being updated (using watch cat )
Current Logstash Configuration
jdbc_user =>{{ include "logstash.database_username" . }}
jdbc_password =>{{ include "logstash.database_password" . }}
schedule => "*/5 * * * * *"
statement => "select id,msg_text from msg_source where id > :sql_last_value"
jdbc_paging_enabled => true
jdbc_page_size => 2000
use_column_value => true
tracking_column_type => "numeric"
tracking_column => "id"
last_run_metadata_path => "/usr/share/logstash/pmw-sql-last-value/{{ .Release.Name }}/messages_sql_last_value.yml"
- I can see that many users are facing the same case.
I really appreciate any help you can provide.