How to rescore Elasticsearch results using semantic search with custom boosting conditions?

Elastic Stack Elasticsearch
1

I am using Elasticsearch's ELSER inference model to perform semantic searches. I need to rescore the search results based on specific conditions: spotlighted documents should be at the top, followed by verified documents, and finally sorted by the highest votes in descending order.

Here is the current Elasticsearch query I am using for semantic search:

GET business-index/_search?filter_path=hits.hits._source.title,hits.hits._source.subtitle,hits.hits._source.categorySearch,hits.hits._source.spotlight,hits.hits._source.verified,hits.hits._source.votes,hits.hits._source.createdAt
{
  "query": {
    "semantic": {
      "query": "hungry",
      "field": "elser"
    }
  }
}

it seems like if i add function_score it does not really take the query into account anymore. I always want the filtered response to take the query into account.
.

"functions": [
        {
          "filter": {
            "term": {
              "spotlight": true
            }
          },
          "weight": 20
        },
        {
          "filter": {
            "term": {
              "verified": true
            }
          },
          "weight": 10
        },
        {
          "field_value_factor": {
            "field": "votes",
            "factor": 1,
            "modifier": "log1p",
            "missing": 0
          },
          "weight": 20
        }
      ],
      "boost_mode": "sum",
      "score_mode": "sum"
    }

Also is there a way to remove some of the weak scores that gets returned from elser field.

If i search hungry i expect to see thing from restaurants and fast foods etc., but i also get results from hair salons and other weird things. i want the aggregate to be on the query as well.

I did look at Reciprocal rank fusion | Elasticsearch Guide [8.17] | Elastic

but not sure how that would help me as i cant figure this out.