Getting some documents ahead of my current time that alter the sql_last_value

Elastic Stack Logstash
1

I get data from a DB every 30 minutes, in the image the current time is 18:50 pm, I should have no data from 18:30 to current time because this data is collected at 19:00 pm

image
image786×268 10.7 KB

It seems that some documents are misconfigured in the DB with a timestamp ahead of time, this also set my sql_last_value ahead of time and when JDBC is collecting data I only get half of it.

To solve this I was thinking in get the current time with ruby and compare it with the timestamp, if timestamp its greater than the current time, drop the event.

if I drop the event the sql_last_value is still recorded in the last run file?

Is there another way, that you may think, to solve this?

2

sql_last_value is stored by the jdbc input. It has no way of knowing whether or not you drop the event, so yes, it is still recorded.

I cannot think of a way to solve this right now.

1 Like
3

If Master Badger can't think of a solution, there is no much hope for me.

the only thing that comes to my mind is to alter the last value with a python script that runs every 25 and 55 minutes in an hour and modifies the value of the epoch timestamp....but Im going to get duplicated values

4

Do you want to exclude the entries from the future entirely or just wait and import them only as soon as “the future is now” for them? If it's the latter, you could just adjust your query condition to timestamp > :sql_last_value AND timestamp <= NOW(), couldn't you?

1 Like
5

:astonished: Great idea Jenni!, Im going to try what you say

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.