Slower geo_distance queries in 2.2

Elastic Stack Elasticsearch
1

I've been using 2.1.1 for the last few months, and a couple days ago upgraded to 2.2.0. With 2.1.1, random continental-US geo_distance queries were taking around 10 ms, with 2.2.0 they are taking more on the order of 150 ms. My test index is about 1.5M documents, all with the geo_point field I am searching against.

I know 2.2.0 is supposed to be faster for geo_point queries, so this is certainly confusing. Just wondering if anyone else was having this experience, or if anyone knew what might be going on. Let me know if any particular info would help.

2

Greetings @PeteyPabPro

random continental-US geo_distance queries... My test index is about 1.5 documents

Can you be more clear? Are you seeing a drastic variance as a function of distance? location? 1.5 what? Thousand, Million? Are your documents multi-valued? Nested? Are you using Queries? Filters? Any additional information you can provide to replicate the performance difference would be helpful.

Thanks!

3

Hi,

Sorry, meant 1.5 million documents (fixed above).

  1. I'm not seeing much variance at all. 136 ms average, with a std dev of 18 over 200 random US lat/long searches.
  2. The field I am filtering is not multi-valued; there are other multi-valued and object fields in the document (none nested, however).
  3. I'm using filters.

I realize this would be hard in any sense to recreate, but really just wanted to see if anyone else was having the same issue.

4

I've definitely encountered this issue as well and are able to recreate it with test data. The following comparisons were performed against v2.1.2, v2.2.0 and 5.0.0 (built from source).

The test data is 6 million documents with random locations on the following index (single shard, query cache disabled). I'm running Elasticsearch with the stock configs on my development machine.

{
  "test": {
    "mappings": {
      "document": {
        "properties": {
          "id": {
            "type": "integer"
          },
          "location": {
            "type": "geo_point",
            "lat_lon": true
          }
        }
      }
    }
  }
}

The geo_distance queries look like:

{
  "query": {
    "bool": {
      "filter": {
        "geo_distance": {
          "distance": "100mi",
          "location": "[lat], [lon]"
        }
      }
    }
  }
}

Here are the average times when performing the same 100 geo_distance searches against the different Elasticsearch versions:

Radius      2.1.2       2.2.0       5.0.0
1mi         256ms       1ms         2ms
10mi        206ms       10ms        2ms
30mi        236ms       36ms        3ms
50mi        252ms       277ms       40ms
100mi       258ms       461ms       74ms
250mi       241ms       2,088ms     315ms
500mi       227ms       3,534ms     643ms
1000mi      223ms       943ms       339ms
2000mi      280ms       754ms       683ms

I was also curious about bounding box performance, so I ran the same queries using a bounding box that circumbscribes the diameter of the geo_distance query:

            2.1.2       2.2.0       5.0.0
1mi         211ms       23ms        9ms
10mi        217ms       72ms        13ms
30mi        221ms       60ms        18ms
50mi        221ms       91ms        23ms
100mi       220ms       89ms        32ms
250mi       223ms       130ms       54ms
500mi       231ms       168ms       81ms
1000mi      231ms       156ms       96ms
2000mi      226ms       143ms       100ms

The slower queries for 2.2.0 and 5.0.0 always have the largest matching count of documents, so queries over areas with lower matching documents perform quickly.

I'm not too sure what to think about the v2.2.0 performance at 250mi and 500mi, but it's definitely not an option for us to run those searches. The next version looks to be an improvement over 2.2.0 but still lags behind 2.1.2 with regards to larger radius searches.

5

I also meant to mention that the average response times follow the same pattern overall even with a smaller set of documents (~500,000).

6

Hi @jweber ,

Awesome analysis - thanks for putting this together. Do you think it's worth filing a bug report?

@nknize what do you think?

7

(reposted from issue 16481)

Hi all,

We are also seeing poor performance with documents indexed on 2.2.1 (as opposed to a preexisting index created on 1.7.3 prior to upgrade).

It looks like this was closed because 2.3.0 was slated to be released sometime in February, but it's been over a month and 2.3.0 hasn't been released, nor can I find any documentation on when it will be released.

Is there a workaround on 2.2 where you can tell Elasticsearch to index using the old format? Or are the only workarounds the script / not upgrading at this time?

(edited for formatting, bad URL)