That's expected as it needs to compute the cosine for every single vector (brute-force). From 30k to 100k means much more vectors.
Using knn should be much faster.
Also look at Script score query | Elasticsearch Guide [8.17] | Elastic
During vector functions' calculation, all matched documents are linearly scanned. Thus, expect the query time grow linearly with the number of matched documents. For this reason, we recommend to limit the number of matched documents with a
queryparameter.