How to run Incremental update(addition of new rows) + updation of existing rows simultaneously, when running SQL query through logstash jdbc plug in

Elastic Stack Logstash
1

Dear Team,

I have a table in oracle DB, which consists of 4 columns, say-- 'Serial_no', 'Transaction_Amt', 'User_Name', 'modification_timestamp'.

Now i run sql select query in logstash conf file(using jdbc connection) and save last serial_no in 'sql_last_value', and save it in 'last_run_metadata_path', with which i see no issues.

Now I add new rows to my table, along with which i update existing row information also. How do i use 'Serial_no' along with 'modification_timestamp' column to retrieve only newly inserted rows and old rows which have an updated data in them.

Regards,
Apeksha

2

@ApekshaB, it sounds like using modification_timestamp instead of Serial_no as your sql_last_value would achieve what you want.

3

But my issue is i want to update existing rows, as well as add new rows in the table simultaneously. I cant simply use modification_timestamp as sql_last_value, because i can have same modification_timestamp for more than one row in the table, in which, query will drop rows with same modification_timestamp in the initial run.
For my JDBC select query i can set sql_last_value either as Serial_no or modification_timestamp. i want to use both of them with a logical OR operator, that means, retrieve rows from table, where sql_last_value > Serial_no OR sql_last_value > modification_timestamp.

How do i achieve this?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.