Geo distance based sorting in Elasticsearch 2.3 stopped working

Elastic Stack Elasticsearch
1

I have implemented search and sorting of users in Elasticsearch based on their location. It used to work fine in Elasticsearch 1.6, but after upgrading to Elasticsearch 2.3.3 it stopped working.

Relevant part of my mapping for user.location in ES is:

{
"user": {
"properties": {
"location": {
"type": "geo_point"
},
}

When I try to execute the following query directly it works fine:

[
{
"from": 0, "size": 3,
"query": {
"bool": { "must_not": { "term": { "id": 2 }
},
"should": { "match_all": {} }
}
},
"sort": [
{
"_geo_distance": {
"user.location": [
{ "lat": 49.2827291, "lon": -123.12073750000002 }
],
"unit": "km"
} } ] } ]

However, when the same query is executed using Java API, there is an exception:

Failed to execute phase [query], all shards failed; shardFailures {[G88txaRXQESUgytUEnTwug][users_zj][0]: RemoteTransportException[[Living Colossus][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"from":0,"size":3,"query":{"bool":{"must_not":{"term":{"id":2}},"should":{"match_all":{}}}},"sort":[{"_geo_distance":{"user.location":[{"lat":49.2827291,"lon":-123.12073750000002}],"unit":"km"}}]}]]; nested: IllegalArgumentException[failed to find mapper for [user.location] for geo distance based sort]; }{[G88txaRXQESUgytUEnTwug][users_zj][1]: RemoteTransportException[[Living Colossus][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"from":0,"size":3,"query":{"bool":{"must_not":{"term":{"id":2}},"should":{"match_all":{}}}},"sort":[{"_geo_distance":{"user.location":[{"lat":49.2827291,"lon":-123.12073750000002}],"unit":"km"}}]}]]; nested: IllegalArgumentException[failed to find mapper for [user.location] for geo distance based sort]; }{[G88txaRXQESUgytUEnTwug][users_zj][2]: RemoteTransportException[[Living Colossus][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"from":0,"size":3,"query":{"bool":{"must_not":{"term":{"id":2}},"should":{"match_all":{}}}},"sort":[{"_geo_distance":{"user.location":[{"lat":49.2827291,"lon":-123.12073750000002}],"unit":"km"}}]}]]; nested: IllegalArgumentException[failed to find mapper for [user.location] for geo distance based sort]; }{[G88txaRXQESUgytUEnTwug][users_zj][3]: RemoteTransportException[[Living Colossus][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"from":0,"size":3,"query":{"bool":{"must_not":{"term":{"id":2}},"should":{"match_all":{}}}},"sort":[{"_geo_distance":{"user.location":[{"lat":49.2827291,"lon":-123.12073750000002}],"unit":"km"}}]}]]; nested: IllegalArgumentException[failed to find mapper for [user.location] for geo distance based sort]; }{[G88txaRXQESUgytUEnTwug][users_zj][4]: RemoteTransportException[[Living Colossus][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"from":0,"size":3,"query":{"bool":{"must_not":{"term":{"id":2}},"should":{"match_all":{}}}},"sort":[{"_geo_distance":{"user.location":[{"lat":49.2827291,"lon":-123.12073750000002}],"unit":"km"}}]}]]; nested: IllegalArgumentException[failed to find mapper for [user.location] for geo distance based sort]; }

Caused by: java.lang.IllegalArgumentException: failed to find mapper for [user.location] for geo distance based sort

This is java code that fires exception:

    Client client = ElasticSearchFactory.getClient();
    QueryBuilder qb = new MatchAllQueryBuilder();
    BoolQueryBuilder bQueryBuilder = QueryBuilders.boolQuery();
    bQueryBuilder.should(qb);

    for (User ignUser : ignoredUsers) {
        if (ignUser != null) {
            bQueryBuilder.mustNot(termQuery("id", ignUser.getId()));
        }
    }
    GeoDistanceSortBuilder sortBuilder = SortBuilders.geoDistanceSort("user.location")
            .point(lat, lon)
            .unit(DistanceUnit.KILOMETERS)
            .order(SortOrder.ASC);

    SearchResponse sResponse = client .prepareSearch(ESIndexNames.INDEX_USERS)
            .setTypes(ESIndexTypes.USER) .setQuery(bQueryBuilder).addSort(sortBuilder)
            .setFrom(0) .setSize(limit).execute() .actionGet();
} catch (NoNodeAvailableException e1) {

Anybody has idea how to investigate and find the problem?

Thanks,
Zoran

2

Hi,

Just some clarification, I'm suprised executing the query directly works, since the keyword for geo distance sort is _geo_distance or _geoDistance. I just tried this on 2.3.3 with a very simple index and it failed because it was assuming there is a field named geodistance it was supposed to sort on. Consequently it failed beacuse neither "user.location" nor "unit" are valid general sort options. Do you also have a field called geodistance that might be used for sorting?

3

Hi Christoph,

Thank you for pointing this. Actually, I just figured out that this "sort" seems to be ignored in direct queries. Whatever I put in place of _geo_distance it returns the same list of users, but it doesn't sort users based on their location. I was using sense and head plugin and each time I get a list of users with location (0,0) even though there are some users with exactly the same location like in the query.

Any idea?

4

Hi Zoran,

works for me, at least with a minimal example given below, when you use the correct _geo_distance keyword for sorting:

DELETE /_all

PUT /index
{
  "mappings": {
    "type": {
      "properties": {
        "id": {
          "type": "string"
        },
        "user": {
          "properties": {
            "location": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  }
}

PUT /index/type/1
{
  "id": "1",
  "user": {
    "location": [
      {
        "lat": 49,
        "lon": -122
      }
    ]
  }
}

PUT /index/type/2
{
  "id": "2",
  "user": {
    "location": [
      {
        "lat": 50,
        "lon": -122
      }
    ]
  }
}

PUT /index/type/3
{
  "id": "3",
  "user": {
    "location": [
      {
        "lat": 51,
        "lon": -122
      }
    ]
  }
}

GET /index/type/_search
{
  "query": {
    "bool": {
      "must_not": {
        "term": {
          "id": 2
        }
      }
    }
  },
  "sort": [
    {
      "_geo_distance": {
        "user.location": [
          {
            "lat": 48,
            "lon": -122
          }
        ],
        "unit": "km"
      }
    }
  ]
}

Gives:

"hits": {
    "total": 2,
    "max_score": null,
    "hits": [
      {
        "_index": "index",
        "_type": "type",
        "_id": "1",
        "_score": null,
        "_source": {
          "id": "1",
          "user": {
            "location": [
              {
                "lat": 49,
                "lon": -122
              }
            ]
          }
        },
        "sort": [
          111.11094781487563
        ]
      },
      {
        "_index": "index",
        "_type": "type",
        "_id": "3",
        "_score": null,
        "_source": {
          "id": "3",
          "user": {
            "location": [
              {
                "lat": 51,
                "lon": -122
              }
            ]
          }
        },
        "sort": [
          333.31414120211184
        ]
      }
    ]
  }
5

Hi Christoph,

I've sent you PM with information about how to access my elasticsearch server. Could you please check it?

Thanks,
Zoran

6

Sorry, I can't work on your server, but please feel free to add any other information here so others can jump in and help.

7

Since my user index type is more complex than basic one you posted, I will post the whole metadata on it here. I would appreciate if you can check on it and see if there is something that might make the problem. Obviously for me sorting on user location is not working in query, but there are results which are not sorted. However, from Java API there is exception related to this user.location mapping, and I guess something is wrong with my mapping, but can't figure out what it could be:

{
  "state": "open",
  "settings": {
    "index": {
      "cluster": {
        "name": "elasticsearch"
      },
      "number_of_shards": "5",
      "http": {
        "enabled": "false"
      },
      "creation_date": "1467695930929",
      "number_of_replicas": "0",
      "uuid": "VaNEsh3USTGJ9hgzpBS9sQ",
      "version": {
        "created": "2030199"
      }
    }
  },
  "mappings": {
    "user": {
      "properties": {
        "system": {
          "type": "boolean"
        },
        "credentials": {
          "type": "nested",
          "properties": {
            "progress": {
              "type": "integer"
            },
            "dateEnrolled": {
              "format": "yyyy/MM/dd HH:mm:ss",
              "type": "date"
            },
            "id": {
              "type": "long"
            },
            "instructorId": {
              "type": "long"
            }
          }
        },
        "roles": {
          "properties": {
            "id": {
              "type": "long"
            }
          }
        },
        "name": {
          "type": "string"
        },
        "location": {
          "type": "geo_point"
        },
        "avatar": {
          "type": "string"
        },
        "id": {
          "type": "long"
        },
        "position": {
          "type": "string"
        },
        "credentialsWithInstructorRole": {
          "properties": {
            "id": {
              "type": "long"
            },
            "dateAssigned": {
              "format": "yyyy/MM/dd HH:mm:ss",
              "type": "date"
            }
          }
        },
        "url": {
          "type": "string"
        },
        "lastname": {
          "type": "string"
        }
      }
    }
  },
  "aliases": []
}
8

Looks like user is your type, and location is defined on the same level as name or id, so your query should be:

{
    "query": {
        "bool": {
            "must_not": {
                "term": {
                    "id": 2
                }
            },
            "should": {
                "match_all": {}
            }
        }
    },
    "sort": [{
        "_geo_distance": {
            "location": [{
                "lat": 49.2827291,
                "lon": -123.12073750000002
            }],
            "unit": "km"
        }
    }]
}

At least thats working for me. Java API or not doesn't make a difference, in 2.3.3 the QueryBuilder and SortBuilder render out Json that is parsed again on the coordinating node, so there is nothing special about the java part that makes it different from going thorugh the REST api.

1 Like
9

Hi Christoph,

Yes. That was the problem.:relaxed:
I don't know why this used to work earlier with older ES, but most important is that it's fixed now.

Thanks a lot for your help

Zoran