Is it possible to set the sql_last_value parameter to go back a set amount of time. So something like :sql_last_value(-10s) or something like that. I'm querying a db and getting anything that's been updated since the last time the query was run. However, it looks like there's some delay going on and it's missing updates.
I have not tested it, but I believe :sql_last_value is just substituted using the value from the file. I cannot see a reason why you should not include standard SQL date math in the query that includes it.
So you think this would work?
Select * from table where column1 > DATEADD(SECOND,-10,:sql_last_value)
I don't see why it wouldn't. Try it out and let us know 
Well, it caused Logstash to completely restart instead of just reloading the one pipeline....and it appears to be working...at least it isn't throwing errors. The logs APPEAR to show it working...
[2021-06-16T16:18:00,201][INFO ][logstash.inputs.jdbc ] (0.007883s) USE DB SELECT * FROM Table WHERE Date > DATEADD(SECOND,-10,'2021-06-16T16:17:30.763')
[2021-06-16T16:18:30,179][INFO ][logstash.inputs.jdbc ] (0.007237s) USE DB SELECT * FROM Table WHERE Date > DATEADD(SECOND,-10,'2021-06-16T16:18:00.169')
I'm seeing updates to the data so it looks like it's working. As always, thanks for the assistance Badger.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.