Too many dynamic script compilations

[elk11]

Elasticsearch and Kibana version: 7.3.1
Number of nodes: 5

In our documents we have a nested fields and since we wanted to create visualizations using those nested fields, we created scripted fields in Kibana management UI using painless. We have around 150 scripted fields in that particular index pattern.

Most of those scripted fields are similar like below:

    if (!doc['key.nestedKey'].empty) {
        return params['_source']['key']['nestedKey']
    }
    return null

And we have increased max_compilation_rate to 1000/1m.

So with all the above configuration, sometimes when filtering on a scripted field from Kibana discover, we get following error:

Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[script] Too many dynamic script compilations within, max: [1000/1m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting","bytes_wanted":0,"bytes_limit":0,"durability":"TRANSIENT"},

So I wanted to get some recommendation on how to approach this issue. When previously we had hit this issue, we just increased the limit. But I guess there's a better way to handle this.

We are also seeing occasional x of y shards failed while doing some Kibana searches. Nothing stands out in elasticsearch logs. Not sure if this is related to same problem.

[Marius_Dragomir]

The main recommendation is to convert those scripted fields into actual fields. There isn't really another way to make them suited for production.
Depending on how much data you have, reindex might be a good solution.
But if you have a lot of data, but only query recent data, ingest pipelines to do the processing needed on your fields before ingest is the best solution.