Field value factor FunctionScore doesn't change score

Elastic Stack Elasticsearch
1

I added FunctionScore with the FieldValueFactor function to my KNN query, but the search hits scores don't get influenced by the numeric value in the fieldValueFactor.field. I can see the scale of the scores changing when I increase fieldValueFactor.factor or FunctionScore.weight.
The same happens with other than FieldValueFactor functions.
Either, I am doing this wrong, or ES doesn't support FunctionScore with the KNN query. My ES server is v8.15.4
Here is the relevant excerpt from the ES query I am sending:
"query":{
"bool":{
"must":{
"function_score":{
"query":{
"knn":{
"field":"vector",
"query_vector":[....],
"filter":[
. . . .
]
}
},
"functions":[{
"field_value_factor":{
"field":"TwitterScore",
"factor":1.2,
"missing":1
}]
. . . . . . .