Elastic Update Script avoid dynamic script compilations

[Siegi (Michael Sieghartsleitner)]

Hey there,
after my elastic usage is growing, I ran into the script.max_compilations_rate(

TransportError(500, 'general_script_exception', '[script] Too many dynamic
script compilations

) in my updates. I use update by query scrips ins my Python application, because I have not saved the document ids outside to use them for updates. An update looks sth like this:

{
            "script": {
                "source": "ctx._source.category_id = params.cat",
                "params": {"cat":2}
            },
            "query": {
                "term": {
                    "feed_id": 164794
                }
            }
        }

I guess I have 3 options here: (correct me if am wrong)

1. update max_compilations_rate: I don't wanna do that for performance reasons
2. Reindex all documents using feed_id as document id and then use "Update API"
3. Change Update Script/method (?)

Is there a way to do this kind of updates(without knowing document id) with indexed scripts/without dynamic scripts?
If yes, where can I find the right documentation?
If no, how can I create a reindex pipeline which uses "feed_id" as doc id(or is this even possibile)?

[spinscale (Alexander Reelsen)]

if that is your script beeing used across all updates, then it is unlikely that the too many dynamic script compilations stems from this script, but from others.

Is this the only script. you are using in your updates? Or just one of many?

[Siegi (Michael Sieghartsleitner)]

This script is used for updates of my Python Application using the update_by_query method of the Elasticsearch Python client. It is performed multiple 100 times per minute with different feed_id 's and different or multiple parameters used for the updates.

I forgot to mention that the script in my question is just one example of the scripts which are used for updating.

[spinscale (Alexander Reelsen)]

The main question here is how many different scripts are in use?

[Siegi (Michael Sieghartsleitner)]

Its mainly the one above and two others(which are search scripts for calculating cosine similaries), but I just saw exceptions from the query above.

{
            "min_score": self._MIN_SCORE,
            "from": offset,
            "size": limit,
            "sort": [
                {"_score": {"order": sorting}}
            ],
            "query": {
                "script_score": {
                    "query": {
                        "bool": {
                            "filter": [
                                {"range": {
                                    "follower_count": {
                                        "gte": min_follower_count,
                                        "lte": max_follower_count
                                    }
                                }
                                }
                            ],
                            "must_not":
                                {
                                    "terms": {"feed_id": exclude}
                                }
                        }
                    },
                    "script": {
                        "source": "cosineSimilarity(params.tv, 'topicvector') + 1.0",
                        "params": {
                            "tv": query_vector,
                        }
                    }
                }
            }, "fields": ["feed_id"], "_source": False
        }

and

{
            "query": {
                "script_score": {
                    "query": {
                        "bool": {
                            "filter": [
                                {
                                    "terms": {
                                        "feed_id": query_feed_ids
                                    }
                                }
                            ]
                        }
                    },
                    "script": {
                        "source": "cosineSimilarity(params.tv, 'topicvector') + 1.0",
                        "params": {
                            "tv": target_vector
                        }
                    }
                }
            }, "fields": ["feed_id"], "_source": True
        }

I also tested my Application without any other scripts (by deactivating all other endpoints which perform other scripts)and the error still appeared.

Could it be that the script above has to be compiled each time it is executed because of different amount of parameters?

[spinscale (Alexander Reelsen)]

You can increase logging for the component that compiles the script to see what is actually compiled. Make sure you disable this again at some point to prevent lots of logging

See Logging | Elasticsearch Guide [7.17] | Elastic

The logger in question would be org.elasticsearch.script.ScriptService.

Hope this helps!