Function Score with Bool Query Producing Wrong Scores

Elastic Stack Elasticsearch
1

I have this filtered function score query, which doesn't work as expected.

I had this functionality working before I upgraded from 1.7 to 2.3.1, I think I am missing something simple with the new syntax after the upgrade.
The documentation says that filtered queries are deprecated so I am using bool queries instead.

If I put the function score query with the function only (no filtering/bool query) it works as expected (closer items will have higher score)

{
  "query": {
    "function_score": {
      "functions": [
        {
          "gauss": {
            "doc.itemCenter": {
              "origin": {
                "lat": 20,
                "lon": 70
              },
              "scale": "500.0mi",
              "offset": "0.0mi",
              "decay": 0.1
            }
          }
        }
      ]
    }
  }
}

But I want to filter the query and also apply the scoring function, so my final query is as below, which is not producing correct results (bool terms are applied but items that are closer are not given higher score)

{
  "query": {
    "function_score": {
      "functions": [
        {
          "gauss": {
            "doc.itemCenter": {
              "origin": {
                "lat": 20,
                "lon": -70
              },
              "scale": "500.0mi",
              "offset": "0.0mi",
              "decay": 0.1
            }
          }
        }
      ],
      "query": {
        "bool": {
          "must": [
            {
              "term": {
                "doc.userId": {
                  "value": "useronetwo"
                }
              }
            },
            {
              "term": {
                "doc.hidden": {
                  "value": false
                }
              }
            }
          ]
        }
      }
    }
  }
}

I also tried to use the filter inside the function "filter": {} but ES ignored the conditions completely.