Is it possible to configure different tracking_column for different jdbc queries in single logstash conf file

Elastic Stack Logstash
1

I have a few queries running via logstash jdbc plugin with tracking_column as a timestamp from db and using sql_last_value while querying. These are set to run every minute. This is working fine.

    use_column_value => true
    tracking_column => time_mark
    schedule=>"* * * * *"
    statement => "SELECT xxxx FROM yyyy WHERE zzzz and time_mark>:sql_last_value"

However, I want to add another query with schedule as once per day. Hence, the sql_last_value metadata file which is getting updated every minute by the current queries cannot be used for this once a day query.

Essentially I want this query to run successfully by getting updated db values only once a day

    use_column_value => true
    tracking_column => time_mark
    schedule=>"0 5 * * *"
    statement => "SELECT aaaa FROM bbbb WHERE cccc and time_mark>:sql_last_value"

What are the other options? Can there be 2 separate tracking columns for 2 different queries? I tested it and it doesn't seem to be supported. How can I set this up? Any suggestions?

2

I think the problem is that both jdbc inputs are saving their state into the same file. Make sure you set the last_run_metadata_path option to unique values for each input.

3

Yes! That was it. Thank you I had missed that parameter while reading up on the jdbc plugin.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.