I have a few queries running via logstash jdbc plugin with tracking_column as a timestamp from db and using sql_last_value while querying. These are set to run every minute. This is working fine.
use_column_value => true tracking_column => time_mark schedule=>"* * * * *" statement => "SELECT xxxx FROM yyyy WHERE zzzz and time_mark>:sql_last_value"
However, I want to add another query with schedule as once per day. Hence, the sql_last_value metadata file which is getting updated every minute by the current queries cannot be used for this once a day query.
Essentially I want this query to run successfully by getting updated db values only once a day
use_column_value => true tracking_column => time_mark schedule=>"0 5 * * *" statement => "SELECT aaaa FROM bbbb WHERE cccc and time_mark>:sql_last_value"
What are the other options? Can there be 2 separate tracking columns for 2 different queries? I tested it and it doesn't seem to be supported. How can I set this up? Any suggestions?