# Database sync using document\_id

**URL:** <https://discuss.elastic.co/t/database-sync-using-document-id/105650>\
**Category:** Logstash\
**Created:** [October 29, 2017, 6:16am UTC](https://discuss.elastic.co/t/database-sync-using-document-id/105650 "2017-10-29T06:16:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![karnamonkster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karnamonkster/32/67266_2.png) [@karnamonkster](https://discuss.elastic.co/u/karnamonkster)\
**Post date:** [October 29, 2017, 6:16am UTC](https://discuss.elastic.co/t/database-sync-using-document-id/105650/1 "2017-10-29T06:16:15Z")

</div>

Hello,

I am trying to sync a database with around 100,000 records.  
Now these records are flags respective to number of devices which keep on updating.

For me the use case is to see the latest status of that device and alarm.

**My configuration:**  
ES & Kibana 5.6.2  
LS 5.4.1  
I am using a JDBC plugin to poll the Oracle Database.  
Scheduled it for every minute to get events from (sysdate - 5minutes).  
Using a **document\_id** for getting unique record **"device\_id"-"device\_name"-"status"**  
But this approach may not get all the device alarms updated hence giving me the old status.

Is there a way to handle this issue in a better way. The goal is not to query the database for all the 100k records, but to update all records in ES without missing any.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 30, 2017, 1:07pm UTC](https://discuss.elastic.co/t/database-sync-using-document-id/105650/2 "2017-10-30T13:07:16Z")

</div>

> But this approach may not get all the device alarms updated hence giving me the old status.

Why is that?

> Is there a way to handle this issue in a better way. The goal is not to query the database for all the 100k records, but to update all records in ES without missing any.

Is there a "last modified" column (or similar) that Logstash can use to determine what has happened since last time?

---

<div class="post-metadata">

**Author:** ![karnamonkster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karnamonkster/32/67266_2.png) [@karnamonkster](https://discuss.elastic.co/u/karnamonkster)\
**Post date:** [October 30, 2017, 4:29pm UTC](https://discuss.elastic.co/t/database-sync-using-document-id/105650/3 "2017-10-30T16:29:29Z")

</div>

No there is no column which has “last modified”.  
That is where I am using document\_id.  
When the record with combination of all three gets a new value is the one last updated. And should be the only record available.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 27, 2017, 4:29pm UTC](https://discuss.elastic.co/t/database-sync-using-document-id/105650/4 "2017-11-27T16:29:58Z")

</div>

Closed on November 27, 2017, 4:29pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.