Questions regarding index.gc_deletes

Elastic Stack Elasticsearch
1

I have a few questions regarding "index.gc_deletes", and I can't seem to find the answers in Elasticsearch documents.

Context:
We have a stream of events to be indexed by Elasticsearch, including "upsert" and "delete" events. We use external versioning to ensure out-of-order events work as intended. However, "delete" events are tricky for us due to "index.gc_deletes", because we want the version of deleted documents to stay forever, so that no documents with a smaller version number can ever be indexed.

Questions:

  1. Would "index.gc_deletes" work as intended if we specify a super large number, say 100 years? Would the versioning info of deleted documents persist "forever" if we do this? Is there any upper bound to how large the index.gc_deletes setting can be?
  2. How would a large "index.gc_deletes" value affect indexing and query performance? I came across this blog from a long time ago. Are the deleted documents in this blog the same as deleted documents related to "index.gc_deletes"?
  3. What happens when a node restarts? Will the versioning info of deleted docs be lost?
  4. Is versioning info of deleted docs persisted with backups, such that they will restore properly?
  5. Is there any upper bound to how many copies of versioning info of deleted docs we can store?
2

index.gc_deletes keeps historical information about deletes in-memory on the node that hosts the corresponding shard (i.e. it is not recorded in Lucene). This means that this info does not survive restarts and can only be seen as "best effort". Setting this setting very high is not desirable as it would eventually have the node run out of memory.

If you want the version of deleted documents to stay forever, it might be better to not issue deletions for these docs, but to instead build your own tombstone mechanism, transforming deletions into updates that set a "deleted" flag, which you then filter on when querying.

3

Thanks for the response @ywelsch!

If you want the version of deleted documents to stay forever, it might be better to not issue deletions for these docs, but to instead build your own tombstone mechanism, transforming deletions into updates that set a "deleted" flag, which you then filter on when querying.

We are considering building such a tombstone mechanism, but we are concerned about the impact on query performance. The way we have thought to build our own tombstoning mechanism is like so:

  • Process "delete" requests by clearing all existing fields on the document and setting a boolean field deleted to true.
  • Add a filtering predicate to all queries to make them exclude documents where deleted is true.

We are concerned about the potential performance impact of that query filter. We are working on indexing a corpus of billions of documents and expects deletes to be extremely rare (probably only a handful of documents). Here's my current mental model of how Elasticsearch would handle the additional !deleted filtering predicate:

  • Elasticsearch will build a set of all ids that satisfy the !deleted predicate. This would contain the ids 99.999% of documents.
  • After applying the other filters, Elasticsearch will perform a set intersection with the set of ids that satisfied the !deleted predicate to get the final filtered list of documents.

This sounds like a very expensive filter to add to all queries. For example, imagine if we add it to a query that is filtering on id to fetch particular documents. That should be a very cheap query (it's just fetching a handful of documents by primary key) but our added !deleted filtering predicate forces Elasticsearch to intersect the small list of document ids with almost the entire corpus of documents....which sounds quite expensive for such a simple query.

Is my mental model here wrong? Is there a way we can implement tombstoning that doesn't cause this kind of performance problem?

4

Here's my current mental model of how Elasticsearch would handle the additional !deleted filtering predicate...

This is not how ES works. There's more info on query/filter execution here: In which order are my Elasticsearch queries/filters executed? | Elastic Blog

The best way to evaluate this is perhaps to build it and try it out. Note that we use similar kind of filtering already internally when you have nested types defined. Nested documents are proper documents in Lucene (but miss a certain internal field), and automatically filtered out of searches based on the absence of that internal field. You could do something similar, only add this "deleted" field to tombstone docs, and then filter them out as documented here.

5

Thanks @ywelsch. I'll dig into those resources.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.