Verifying data consistency between Oracle and ES

Elastic Stack Elasticsearch
1

I have an Oracle database with 80 million records. These have been migrated to elasticsearch and there is a continuous sync running that will keep the data updated in elasticsearch whenever data is added/changed in the Oracle database. I have two questions regarding this.

  1. How to verify if all 80 million records have been indexed in ES? This might be fairly easy running count
  2. How to verify the data consistency on a periodic basis, say each week to ensure that both data sources, Oracle and ES are in sync?
2

It depends. If you have for example the same id in Oracle and elasticsearch and also a timestamp for example, you can run a count per year in your database and an aggregation in elasticsearch and compare the number of documents per year. Then if you have a difference, you can compare months...
Then compare the ids of a month in the database with the ones in elasticsearch.

This is basically what I did in the past for a similar problem.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.