ES operations and memory usage

Elastic Stack Elasticsearch
1

I am struggling a bit to understand which ES operations use up which part
of memory

  • From reading the documentation I know that faceting and sorting take up
    heap memory and also that some filters are cached in JVM heap. However,
    what other operations take up JVM heap memory?
  • When faceting on a particular field are all of the values of that field
    loaded into JVM heap or only the distinct values of the field. I would
    believe it is the latter as fields of high cardinality require more JVM
    heap.
  • What operations use OS memory. I believe documents retrieved and ES index
    are kept in OS memory. Can anyone confirm this? What other operations go to
    OS memory?

Finally, I would also be interested if there are any posts or tutorials
that desribce the complete lifecycle of a query from submission of query to
return of resultset.

Thanks
Uli

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/561712ab-e11b-454f-b95e-38612cbb8db1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2
  • Fielddata (sorting, facets, some scripting), filter cache (filters), and
    id_cache (parent-child) are probably the ones that would affect memory
    usage significantly.
  • Fielddata can be loaded in different ways depending on data type and
    other config (details here:
    Elasticsearch Platform — Find real-time answers at scale | Elastic)
  • OS memory would generally be file system caches for the underlying Lucene
    indexes

About your other question, the default query execution is called
query_then_fetch
(Elasticsearch Platform — Find real-time answers at scale | Elastic).
The basic idea is the search (query) is scattered in parallel to all shards
(single replica set) of an index being searched. Then a little bit of
results are gathered back, reduced, and then sorted. Then whatever is the
final reduced list, ES goes back to the final shards and pulls out (fetch)
additional data required to return final hits to the caller.

On Wednesday, March 26, 2014 10:41:44 AM UTC-4, Uli Bethke wrote:

I am struggling a bit to understand which ES operations use up which part
of memory

  • From reading the documentation I know that faceting and sorting take up
    heap memory and also that some filters are cached in JVM heap. However,
    what other operations take up JVM heap memory?
  • When faceting on a particular field are all of the values of that field
    loaded into JVM heap or only the distinct values of the field. I would
    believe it is the latter as fields of high cardinality require more JVM
    heap.
  • What operations use OS memory. I believe documents retrieved and ES
    index are kept in OS memory. Can anyone confirm this? What other operations
    go to OS memory?

Finally, I would also be interested if there are any posts or tutorials
that desribce the complete lifecycle of a query from submission of query to
return of resultset.

Thanks
Uli

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dbe09764-67e8-49cd-98e7-9c9b5e4a801a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.