In preparation for upgrading our Prod environment, we've built up our Dev environment to have more processors and memory and are testing out our heavy watcher load to see if all is well there before doing the same upgrade to processors/memory in Prod.
As of now, we have 325+ watchers running in prod every 2 minutes; in Dev that number is closer to 250.
The problem is that in Dev, I'm seeing the circuit breaking exception:
[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting.
Both our Dev & Prod settings for this:
Our script.max_compilations_rate: 75/5m
Prod doesn't get these errors at all. Dev is 3 nodes and Prod is 6 nodes, though Prod has less processors (1 per node) than Dev (4), so I would expect Prod to have these problems.
Does anybody know what I should check next?