Need help in understanding script - compilations from _nodes/stats

Karthik_Ramachandran · December 15, 2018, 12:23am

Hi All,

We are getting circuit breaker exceptions on script compilations during watch deployment, and after increasing from default 75/5m to a higher value viz. 200/5s, it worked. But, wanted to check what is the exact number of scripts that were running given a time range. Used _nodes/stats to check the state of nodes , which results something like below for each node

"script": {
"compilations": <Varied number for each node ranging from 2000 - 25000>,
"cache_evictions": <Varied number for each node ranging from 2000 - 25000>
}

I'm not sure how to read this. If it is 75/5m it should be failing for every deployment and every watch that is running behind. Any help in reading and understand how to get current compilations/minute would help.

NOTE: The CPU usage was less so we did not have any issue after increasing compilation rate.

DavidTurner · December 16, 2018, 9:12am

These numbers are cumulative totals. To work out the rate at which these things occur, you need to take two readings some time apart and subtract one from the other. The derivative aggregation is one way to do this.

Karthik_Ramachandran · December 17, 2018, 5:49pm

Thanks David. Given that it is cumulative, the first time when I use stats API, is it the number shown is from the time of cluster-node start. Please correct me.

DavidTurner · December 17, 2018, 6:14pm

Yes (in the master branch anyway, I haven't checked all versions). It is zero when the node starts, increases by 1 on each compilation, and is never reset to zero:

github.com

elastic/elasticsearch/blob/99f88f15c5febbca2d13b5b5fda27b844153bf1a/server/src/main/java/org/elasticsearch/script/ScriptMetrics.java#L24-L44


      
          public class ScriptMetrics {
              final CounterMetric compilationsMetric = new CounterMetric();
              final CounterMetric cacheEvictionsMetric = new CounterMetric();
              final CounterMetric compilationLimitTriggered = new CounterMetric();
          
              public ScriptStats stats() {
                  return new ScriptStats(compilationsMetric.count(), cacheEvictionsMetric.count(), compilationLimitTriggered.count());
              }
          
              public void onCompilation() {
                  compilationsMetric.inc();
              }
          
              public void onCacheEviction() {
                  cacheEvictionsMetric.inc();
              }
          
              public void onCompilationLimit() {
                  compilationLimitTriggered.inc();
              }
          }

system · January 14, 2019, 6:14pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unfounded script recompilations Elasticsearch	7	684	September 4, 2019
Unkown script compilations for template-context in script-cache Elasticsearch	1	212	March 16, 2023
ElasticSearch inine script, max_compilation_rate issue Elasticsearch	3	544	July 21, 2020
Upping the value of set script.max_compilations_per_minute Elasticsearch	3	3428	March 28, 2017
Werid err: org..CircuitBreakingException: [script] Too many dynamic script compilations Elasticsearch	4	583	June 18, 2020

Need help in understanding script - compilations from _nodes/stats

Related topics