Elasticsearch aggregation OOM

Elastic Stack Elasticsearch
21

Yes, it does.

The data associated with doc values is always on disk. However for certain types of docvalues there are small lookups (partly) in the jvm that contain file offsets for where to start looking for the actual data. The doc_values_memory_in_bytes stat reports the size of these lookups.

Not directly. For fields with doc values enabled, the size of field data cache only includes the size of global ordinals. For fields that rely on field data (for example analyzed strings), as far as I know there is no good way to determine what the size of global ordinals are, as the field data size will include field data itself.

In case for doc values fields with eager loading, only global ordinals will be loaded. In case for fields configured to use field data (by default that are analyzed string fields), field data has to be loaded. Global ordinals in that case depend on field data, without it can't be built.

1 Like
22

@mvg Thanks for the clarifications, this clears most things.
I am still seeing a weird behaviour wrt to my last point.

When I update the mapping to

{
  'type' => 'string',
  'index' => 'not_analyzed',
  'doc_values' => true,
  'fielddata' => array(
    'format' => 'disabled',
    'loading' => 'eager_global_ordinals',
  )
}

and start indexing documents _cat/fielddata reports 0b being used even after I issue flushes periodically. And aggregations still failed stating fielddata is disabled.

However when I remove 'format' => 'disabled' and repeat the process _cat/fielddata does show memory being consumed.
If global ordinals are being built using doc values then shouldn't aggregations have worked with fielddata being disabled ?

23

By disabling field data the entire cache is disabled and thus global ordinals can't be stored either. So I guess that is why you don't see any memory being consumed. I haven't verified, but disabling field data would not make any of your search requests work even though doc values have been configured?

24

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.