I am trying to sync a database with around 100,000 records.
Now these records are flags respective to number of devices which keep on updating.
For me the use case is to see the latest status of that device and alarm.
ES & Kibana 5.6.2
I am using a JDBC plugin to poll the Oracle Database.
Scheduled it for every minute to get events from (sysdate - 5minutes).
Using a document_id for getting unique record "device_id"-"device_name"-"status"
But this approach may not get all the device alarms updated hence giving me the old status.
Is there a way to handle this issue in a better way. The goal is not to query the database for all the 100k records, but to update all records in ES without missing any.