Also came across this thread:
In my measurement where I bumped up the max_compilations_rate manually via persistent cluster settings, I noticed significant degradation in update latency. In my test, I am running 10000 updates/minute and all 10000 push unique painless script.
Also, I am using elasticsearch 7.5 and the I was not able to change cache_size or cache_expire dynamically. I was able to change only max_compilations_rate setting. These attributes are context based in later releases of elasticsearch, but we are not planning to move to later versions of easlticsearch at the moment.
We are concluding that the cache eviction/reload comes with a compilation overhead even for stored scripts. One logical explanation I could think of is - if the stored script was not cached, and was updated in the backend, then they have to be recompiled before loading them into cache again. That's the only explanation I could think of for recompiling stored script every time when we reload them into cache.
Compilation overhead results into much slower update, impacting update latency by an order of magnitude.