I was able to slow down the query to around 600ms. The way I did it was by indexing lots of documents with overlapping envelopes. Then my point query is hitting ~3million documents and that is lowing down the query.
How many documents is your query matching?