After I completed the analysis of the fielddata cache in my situation, I
sent a longer analysis within Microsoft. A colleague pointed me to the
page
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html
.
There is lots of great info there and I'm really glad that the default size
of the fielddata cache is clearly called out as unbounded.
There are two serious flaws with that page that I request get fixed.
First, regarding this paragraph:
The indices.fielddata.cache.size controls how much heap space is allocated
to fielddata. When you run a query that requires access to new field
values, it will load the values into memory and then try to add them to
fielddata. If the resulting fielddata size would exceed the specified size,
then other values would be evicted in order to make space.
The last sentence is true ONLY when all of the 16 segments in the cache are
filled up. In reality the total size is divided by 16 and then checked on
a segment by segment basis. I have seen posts complaining that they set
the fielddata cache size and see evictions even though the cache as a whole
isn't filled up to the size they configured.
The evictions occur on a segment basis. If you see evictions even though
the cache isn't 'full', your use of segments is not evenly spread across
all 16 -- not that there is anything you can do about that! I suspect that
as ES deployments grow in scale the 16 segments would become more evenly
used.
Second, there are the lines:
There is another setting which you may see online:
indices.fielddata.cache.expire
We beg that you never use this setting! It will likely be deprecated in
the future.
This setting tells Elasticsearch to evict values from fielddata if they
are older than expire, whether the values are being used or not.
I can imagine that the setting might not be a good idea or might even be
deprecated, but the reason given is wrong. The expire value and the
expiration process in the cache definitely compare to last accessed time --
which is updated every time the value is accessed. Thus if the values are
being accessed within the expiry window, they will NOT be evicted.
Craig.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/34fceb06-8df8-4ba4-83bf-2969f4561732%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.