[Announce] Scrutineer for ElasticSearch - Detecting inconsistencies in your data


(Paul Smith) #1

Just wanted to let everyone know that we (Aconex) have open-sourced an ASL2
licensed utility to look for any inconsistencies between the information
stored in ElasticSearch and a JDBC database (with extension points for
other sources).

You can find the project here: https://github.com/Aconex/scrutineer

First off: This tool was NOT developed because ElasticSearch is buggy,
NOOOO, this is because can happen. If you rely on the data
stored in elasticsearch being accurate (that is, your client application is
sending it the right info) than Scrutineer can help. For those
applications where Near-Real-Time (NRT) indexing is part of the use case,
then Scrutineer is very useful.

Scrutineer compares the Version property stored in your ElasticSearch
record with a matching one from your source-of-truth (say DB) and reports
inconsistent state and missing records, relying on you indexing this
Version property using the VersionType.EXTERNAL flag.

Scrutineer can be used in many cases where a full reindex would be very
costly, such as:

  • Detecting and reindexing 1-50% of your index may be quicker than
    performing a full reindex. Once the error rate approaches >50% it may just
    be quicker to reindex though. Your mileage may vary.

  • Reindex preparation - You could prepare a new copy of your index on your
    cluster (or a different one) using a snapshot of your production DB (to
    minimise adverse load/disruptions to production customers), then use
    Scrutineer to quickly find and index what's changed since that DB snapshot
    then switch your index alias to the new one.

  • "Filesystem" scrubber. Run regularly in production to find and hand off
    to another tool to fix any errors before it affects your customers.

  • Disaster Recovery - if you copy a gateway snapshot to a DR location
    periodically to keep closely in sync with your DB, Scrutineer can be used
    at Disaster time (heaven forbid) to make sure your index is in sync with
    your recovered DB.

  • If you ever had to take your database 'back in time' due to a
    catastrophic failure you could recover your elasticsearch indexstate
    quickly by 'rollingback' changes; deleting records that are no longer in
    the db and freshening stale entries back to their earlier states. This may
    be much faster than a full reindex depending on how far back you have to go.

Scrutineer is pretty fast, for reference it took 3.5 seconds to verify 275k
records from a 2-node ES cluster (with vanilla config) against a database
table.

There's a tarball download here:

If you have any questions, please let us know.

cheers,

Paul Smith


(David Pilato) #2

Hi Paul,

That's a really nice tool you gave to us !
I will play with it this week.

Is there any way to bypass the version check ?
My use case doesn't need version check.

If not, I will have a look and try to submit a pull request.

Cheers,
David :wink:
@dadoonet

Le 14 nov. 2011 à 02:50, Paul Smith tallpsmith@gmail.com a écrit :

Just wanted to let everyone know that we (Aconex) have open-sourced an ASL2 licensed utility to look for any inconsistencies between the information stored in ElasticSearch and a JDBC database (with extension points for other sources).

You can find the project here: https://github.com/Aconex/scrutineer

First off: This tool was NOT developed because ElasticSearch is buggy, NOOOO, this is because can happen. If you rely on the data stored in elasticsearch being accurate (that is, your client application is sending it the right info) than Scrutineer can help. For those applications where Near-Real-Time (NRT) indexing is part of the use case, then Scrutineer is very useful.

Scrutineer compares the Version property stored in your ElasticSearch record with a matching one from your source-of-truth (say DB) and reports inconsistent state and missing records, relying on you indexing this Version property using the VersionType.EXTERNAL flag.

Scrutineer can be used in many cases where a full reindex would be very costly, such as:

  • Detecting and reindexing 1-50% of your index may be quicker than performing a full reindex. Once the error rate approaches >50% it may just be quicker to reindex though. Your mileage may vary.

  • Reindex preparation - You could prepare a new copy of your index on your cluster (or a different one) using a snapshot of your production DB (to minimise adverse load/disruptions to production customers), then use Scrutineer to quickly find and index what's changed since that DB snapshot then switch your index alias to the new one.

  • "Filesystem" scrubber. Run regularly in production to find and hand off to another tool to fix any errors before it affects your customers.

  • Disaster Recovery - if you copy a gateway snapshot to a DR location periodically to keep closely in sync with your DB, Scrutineer can be used at Disaster time (heaven forbid) to make sure your index is in sync with your recovered DB.

  • If you ever had to take your database 'back in time' due to a catastrophic failure you could recover your elasticsearch indexstate quickly by 'rollingback' changes; deleting records that are no longer in the db and freshening stale entries back to their earlier states. This may be much faster than a full reindex depending on how far back you have to go.

Scrutineer is pretty fast, for reference it took 3.5 seconds to verify 275k records from a 2-node ES cluster (with vanilla config) against a database table.

There's a tarball download here: https://github.com/Aconex/scrutineer/downloads

If you have any questions, please let us know.

cheers,

Paul Smith


(Paul Smith) #3

Just set your version propert in ES to 0 then and return a literal 0
from the db query for the 2nd column. That should work.

On Monday, November 14, 2011, David Pilato david@pilato.fr wrote:

Hi Paul,

That's a really nice tool you gave to us !I will play with it this week.
Is there any way to bypass the version check ?My use case doesn't need version check.
If not, I will have a look and try to submit a pull request.
Cheers,
David ;-)@dadoonet
Le 14 nov. 2011 à 02:50, Paul Smith tallpsmith@gmail.com a écrit :

Just wanted to let everyone know that we (Aconex) have open-sourced an ASL2 licensed utility to look for any inconsistencies between the information stored in ElasticSearch and a JDBC database (with extension points for other sources).

You can find the project here: https://github.com/Aconex/scrutineer

First off: This tool was NOT developed because ElasticSearch is buggy, NOOOO, this is because can happen. If you rely on the data stored in elasticsearch being accurate (that is, your client application is sending it the right info) than Scrutineer can help. For those applications where Near-Real-Time (NRT) indexing is part of the use case, then Scrutineer is very useful.

Scrutineer compares the Version property stored in your ElasticSearch record with a matching one from your source-of-truth (say DB) and reports inconsistent state and missing records, relying on you indexing this Version property using the VersionType.EXTERNAL flag.

Scrutineer can be used in many cases where a full reindex would be very costly, such as:

  • Detecting and reindexing 1-50% of your index may be quicker than performing a full reindex. Once the error rate approaches >50% it may just be quicker to reindex though. Your mileage may vary.

  • Reindex preparation - You could prepare a new copy of your index on your cluster (or a different one) using a snapshot of your production DB (to minimise adverse load/disruptions to production customers), then use Scrutineer to quickly find and index what's changed since that DB snapshot then switch your index alias to the new one.

  • "Filesystem" scrubber. Run regularly in production to find and hand off to another tool to fix any errors before it affects your customers.

  • Disaster Recovery - if you copy a gateway snapshot to a DR location periodically to keep closely in sync with your DB, Scrutineer can be used at Disaster time (heaven forbid) to make sure your index is in sync with your recovered DB.

  • If you ever had to take your database 'back in time' due to a catastrophic failure you could recover your elasticsearch indexstate quickly by 'rollingback' changes; deleting records that are no longer in the db and freshening stale entries back to their earlier states. This may be much faster than a full reindex depending on how far back you have to go.

Scrutineer is pretty fast, for reference it took 3.5 seconds to verify 275k records from a 2-node ES cluster (with vanilla config) against a database table.
There's a tarball download here: https://github.com/Aconex/scrutineer/downloads

If you have any questions, please let us know.
cheers,
Paul Smith


(Shay Banon) #4

Great job Paul!, thanks for sharing this with the community. The effort of
making your own in house project open source comes with a cost, thanks for
going through it :slight_smile:

On Mon, Nov 14, 2011 at 3:50 AM, Paul Smith tallpsmith@gmail.com wrote:

Just wanted to let everyone know that we (Aconex) have open-sourced an
ASL2 licensed utility to look for any inconsistencies between the
information stored in ElasticSearch and a JDBC database (with extension
points for other sources).

You can find the project here: https://github.com/Aconex/scrutineer

First off: This tool was NOT developed because ElasticSearch is buggy,
NOOOO, this is because can happen. If you rely on the data
stored in elasticsearch being accurate (that is, your client application is
sending it the right info) than Scrutineer can help. For those
applications where Near-Real-Time (NRT) indexing is part of the use case,
then Scrutineer is very useful.

Scrutineer compares the Version property stored in your ElasticSearch
record with a matching one from your source-of-truth (say DB) and reports
inconsistent state and missing records, relying on you indexing this
Version property using the VersionType.EXTERNAL flag.

Scrutineer can be used in many cases where a full reindex would be very
costly, such as:

  • Detecting and reindexing 1-50% of your index may be quicker than
    performing a full reindex. Once the error rate approaches >50% it may just
    be quicker to reindex though. Your mileage may vary.

  • Reindex preparation - You could prepare a new copy of your index on your
    cluster (or a different one) using a snapshot of your production DB (to
    minimise adverse load/disruptions to production customers), then use
    Scrutineer to quickly find and index what's changed since that DB snapshot
    then switch your index alias to the new one.

  • "Filesystem" scrubber. Run regularly in production to find and hand off
    to another tool to fix any errors before it affects your customers.

  • Disaster Recovery - if you copy a gateway snapshot to a DR location
    periodically to keep closely in sync with your DB, Scrutineer can be used
    at Disaster time (heaven forbid) to make sure your index is in sync with
    your recovered DB.

  • If you ever had to take your database 'back in time' due to a
    catastrophic failure you could recover your elasticsearch indexstate
    quickly by 'rollingback' changes; deleting records that are no longer in
    the db and freshening stale entries back to their earlier states. This may
    be much faster than a full reindex depending on how far back you have to go.

Scrutineer is pretty fast, for reference it took 3.5 seconds to verify
275k records from a 2-node ES cluster (with vanilla config) against a
database table.

There's a tarball download here:
https://github.com/Aconex/scrutineer/downloads

If you have any questions, please let us know.

cheers,

Paul Smith


(system) #5