ES operations and memory usage


(Uli Bethke) #1

I am struggling a bit to understand which ES operations use up which part
of memory

  • From reading the documentation I know that faceting and sorting take up
    heap memory and also that some filters are cached in JVM heap. However,
    what other operations take up JVM heap memory?
  • When faceting on a particular field are all of the values of that field
    loaded into JVM heap or only the distinct values of the field. I would
    believe it is the latter as fields of high cardinality require more JVM
    heap.
  • What operations use OS memory. I believe documents retrieved and ES index
    are kept in OS memory. Can anyone confirm this? What other operations go to
    OS memory?

Finally, I would also be interested if there are any posts or tutorials
that desribce the complete lifecycle of a query from submission of query to
return of resultset.

Thanks
Uli

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/561712ab-e11b-454f-b95e-38612cbb8db1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Binh Ly-2) #2

About your other question, the default query execution is called
query_then_fetch
(http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-search-type.html#query-then-fetch).
The basic idea is the search (query) is scattered in parallel to all shards
(single replica set) of an index being searched. Then a little bit of
results are gathered back, reduced, and then sorted. Then whatever is the
final reduced list, ES goes back to the final shards and pulls out (fetch)
additional data required to return final hits to the caller.

On Wednesday, March 26, 2014 10:41:44 AM UTC-4, Uli Bethke wrote:

I am struggling a bit to understand which ES operations use up which part
of memory

  • From reading the documentation I know that faceting and sorting take up
    heap memory and also that some filters are cached in JVM heap. However,
    what other operations take up JVM heap memory?
  • When faceting on a particular field are all of the values of that field
    loaded into JVM heap or only the distinct values of the field. I would
    believe it is the latter as fields of high cardinality require more JVM
    heap.
  • What operations use OS memory. I believe documents retrieved and ES
    index are kept in OS memory. Can anyone confirm this? What other operations
    go to OS memory?

Finally, I would also be interested if there are any posts or tutorials
that desribce the complete lifecycle of a query from submission of query to
return of resultset.

Thanks
Uli

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dbe09764-67e8-49cd-98e7-9c9b5e4a801a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #3