What is in Segments Memory?

Elastic Stack Elasticsearch
1

###What is in segments memory?

GET _stats/?pretty&human
{
 "segments" : {
    "count" : 160,
    "memory" : "24.5gb",
  } ...
   
}

I thought segments were stored as files, cached by the OS, and did not reside in ES's heap. However, that does not seem to be the case. The documentation only says that "[s]egments need to store some data into memory in order to be searchable efficiently." What is this "some data"?

2

There is an in-memory terms index, stored as an FST, that Lucene uses to quickly determine which on-disk block might contain a requested term.

Dimensional points for numeric range filters (coming in 5.0) also has a small in memory tree structure.

Deleted documents bitset (only if there are deletions).

Doc values and stored fields codec formats also have small in-memory structures.

Recent versions of ES break out more details if you pass &verbose=true to the Indices Segments API:

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-segments.html

Mike McCandless

2 Likes