Apply function score after filter

I would like to apply function score only after a regular filter.

Is this somehow possible?
So far it looks like I can only specify one query (the function score query).

I have been trying to hack this with a min_score of 1 and "first" function score filters that return 0 for what I do not want, but it just doesn't seem right.