Question about Segment described in "Elasticsearch: The Definitive Guide"

Elastic Stack Elasticsearch
1

In the section "near real time search" of the book "Elasticsearch: The Definitive Guide":
https://www.elastic.co/guide/en/elasticsearch/guide/current/near-real-time.html

The Figure 20 depicts how in-memory buffer contents been re-freshed into segment in filesystem cache.

The the figure, the commit point includes three GREEN segment, but not include that gray segment in filesystem cache.

My understanding is, the commit point in this situation should include the gray segment as well, since the commit point lists all known segments which are searchable and the gray segment in filesystem cache is searchable now.

Am I right?

2

I think that is supposed to represent the previous commit point.

3

Thanks, so after the refresh, the commit point SHOULD include all green and gray segments, right?

4

That's how I read it, yes.

5

After further reading, I think I might be wrong.
per below statement in https://www.elastic.co/guide/en/elasticsearch/guide/current/translog.html

Elasticsearch uses the commit point during startup or when reopening an index to decide which segments belong to the current shard.

So if the commit point includes the uncommitted segment in filesystem cache, it will cause problem if:

  1. Node is down before filesystem cache is flushed into disk,
  2. Node restarted again

In this case, the commit point will include an non-existing segment...

Then comes to the question, if commit point is not updated after re-fresh, how does ES knows which doc is updated/deleted if the updated/deleted document is still in uncommitted segment in filesystem cache, since the .del file is included in commit segment?

6

The gray segment does not represent a previous commit as previously suggested. It is not included in the commit because a commit operation has not been run since the segment has been flushed to disk. In the previous diagram there are three segments which have been flushed to disk AND committed. when indexing continues the in memory buffer will be filled up with new documents which at some point (either through an explicit or implicit operation) is flushed to disk creating a new segment (the grey segment). This segment is now searchable but is NOT committed yet as the segment as not been f-synced to disk so if the node was to fail now the segment files for the grey segment would not have been persisted and would not survive after a restart. Once the commit operation is called the grey segment will be f-synced, and a commit point will be created which includes this new grey segment. So the commit point will never include uncommited (or non-fsynced segments)

On a re-refresh the IndexReader used to read the segments is updated to include the new uncommitted segment (this is why the segment is referred to in the documentation as searchable but uncommitted). If the node was to fail and restart that segment would be lost since it is uncommitted, but the data from that segment would be replayed from the translog as part of the startup sequence so the data was not lost from Elasticsearch entirely.

2 Likes
7

Thanks for clarifying :slight_smile:

8

Thanks detailed explanation.

Then comes another question:)

Per following statement from https://www.elastic.co/guide/en/elasticsearch/guide/current/dynamic-indices.html
"Segments are immutable, so documents cannot be removed from older segments, nor can older segments be updated to reflect a newer version of a document. Instead, every commit point includes a .del file that lists which documents in which segments have been deleted."

Seems the .del file is included in commit point. So if the new deleted/updated documents are refreshed into GRAY segment and becomes searchable (but not committed yet), is the .del file updated so that ES knows what documents are deleted after fetching documents from all segments (both green and gray) ?

If .del file is not updated till commit point is updated, from where ES knows the document is deleted?

9

.del files like all other segment files are immutable and therefore never updated. when the deleted/updated documents are refreshed a new .del file will be flushed to disk (but not f-synced) and the IndexReader will be updated to include the new segments (which includes the new .del file as this is part of that segment).

The commit point is simply a list of the segments which are part of the index. If this commit point is not written (e.g. because of a node failure) the index will not contain the documents (and deletions) in the segment and upon restart the index will represent the state before the uncommitted segment(s) were flushed. Again this is where the translog comes into play and will replay all operations received by Elasticsearch between the last successful commit point and the failure.

1 Like