Elasticsearch polygon filter performance

I have been playing around with the polygon filter on our index and with the desired accuracy of the polygon on 2 million documents i'm getting search times of 200-300ms at best. I've yet to add a bounding box filter as I hear that will speed things up a bit but my question is, will adding a load of nodes speed things up? I really need to get under 100ms.

This is for a high traffic site, presumably if we start doing polygon searches like this is it likely that the cluster performance as a whole will suffer badly as well? Also, what is the point of setting geohash on the geo_point type, is this just so you can search against the geohash value yourself or does it affect performance in any way?

Loving the geospatial stuff, we just need to fiddle a bit more before it's production useable.