Database sync using document_id

Elastic Stack Logstash
1

Hello,

I am trying to sync a database with around 100,000 records.
Now these records are flags respective to number of devices which keep on updating.

For me the use case is to see the latest status of that device and alarm.

My configuration:
ES & Kibana 5.6.2
LS 5.4.1
I am using a JDBC plugin to poll the Oracle Database.
Scheduled it for every minute to get events from (sysdate - 5minutes).
Using a document_id for getting unique record "device_id"-"device_name"-"status"
But this approach may not get all the device alarms updated hence giving me the old status.

Is there a way to handle this issue in a better way. The goal is not to query the database for all the 100k records, but to update all records in ES without missing any.

2

But this approach may not get all the device alarms updated hence giving me the old status.

Why is that?

Is there a way to handle this issue in a better way. The goal is not to query the database for all the 100k records, but to update all records in ES without missing any.

Is there a "last modified" column (or similar) that Logstash can use to determine what has happened since last time?

3

No there is no column which has “last modified”.
That is where I am using document_id.
When the record with combination of all three gets a new value is the one last updated. And should be the only record available.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.