Using distanceInMiles to find distance fails

Elastic Stack Elasticsearch
1

Using ES 2.3.3
A section of my document mapping is as follows :

"rents": {
  "type": "nested",
  "properties": {.........e.g. "avg"}
},
"location": {
    "type": "geo_point"
}

And in groovy script, I try to find distance of this field

 {
  "nested": {
    "path": "rents",
    "query": {
      "function_score": {
        "query": {...some rents queries....},
        "functions": [
          { 
            "script_score": {
              "params": {
                "center_lat": 23.509518,
                "center_long":-18.378842
              },
              "script": "return doc['location'].distanceInMiles(center_lat, center_long)*rents.avg;"
            }
          }
        ]
      }
    }
  }
}

but this keeps throwing error:

"reason": {
  "type": "script_exception",
  "reason": "failed to run inline script [return doc['location'].distanceInMiles(center_lat, center_long)*rents.avg;] using lang [groovy]",
  "caused_by": {
    "type": "null_pointer_exception",
    "reason": null
  }
}

I am following the guide https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-scripting.html#_document_fields then what am I doing wrong? Please note all my documents have location field, none of them are null or empty. It seems doc is not accessible here

2

You're looking at a Lucene Expression example (here), not Groovy.
Also note the rents.avg part of the expression.

3

@ywelsch What is the solution you suggest?

4

And fyi, even if I remove *rents.avg that doesn't help. Somehow it cannot get access to doc, is it because I have path: rents at the top?

5

Is there a stack trace in the logs?

can you try using _source to access the field?

6

I don't want to use _source as it will be much slower than doc

7

can you at least test if it works with _source?

8

I tried with _source and even that did not work :frowning: also I removed *rents.avg in my script. This is the log

[2016-10-28 09:38:32,418][DEBUG][action.search            ] [Franklin Richards] [listings_v1][0], node[9qsipmkpQWGVH6Rm8zQMUg], [R], v[57], s[STARTED], a[id=tShMmgAFSnW1bRVA3AxJRw]: Failed to execute [org.elasticsearch.action.search.SearchRequest@5f6c6e5e]
RemoteTransportException[[Blind Justice][127.0.0.1:9301][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: ScriptException[failed to run inline script [return _source['location'].distanceInMiles(center_lat, center_long);] using lang [groovy]]; nested: NullPointerException[Cannot invoke method distanceInMiles() on null object];
Caused by: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: ScriptException[failed to run inline script [return _source['location'].distanceInMiles(center_lat, center_long);] using lang [groovy]]; nested: NullPointerException[Cannot invoke method distanceInMiles() on null object];
  at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:409)
  at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)
  at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
  at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:378)
  at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
  at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
  at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
  at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
  at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300)
  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
9

Hi,

I got interested in this problem as well and tried this on a stripped down example. I also get the NPE, the Stacktrace on ES 2.4.1 looks like

[...]
Caused by: java.lang.NullPointerException
    at org.elasticsearch.index.fielddata.ScriptDocValues$GeoPoints.distanceInMiles(ScriptDocValues.java:345)
    at ccc06c13684b2df15349c90529c18c72f105c627.run(ccc06c13684b2df15349c90529c18c72f105c627:1)
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript$1.run(GroovyScriptEngineService.java:281)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:278)

which (after looking at the code) most likely means the geopoint information isn't there. This is probably the case because the script is run in the "nested" scope. I don't know yet if this is expected or avoidable, haven't found much information about acessing the "parent" docs fielddata like that yet.

One way of trying to circumvent the problem for now might be to use scripted sorting on the top level of the query, something like:

{
  "query": {    
     ...  
  },
  "sort": {
    "_script": {
      "type": "number",
      "script": {
        "lang": "groovy",
        "params": {
          "center_lat": 23.509518,
          "center_long": -18.378842
        },
        "inline": "doc['location'].distanceInMiles(center_lat, center_long)"
      },
      "order": "desc"
    }
  }
}

but I haven't found a way to include the " * rents.avg" multiplier with this approach yet. But maybe this helps a bit. Will keep digging.

10

Thanks @cbuescher for the suggested workaround. Unfortunately, that will not help me because I won't get access to * rents.avg

I am now changing the mapping to further de-normalize my data and include same location field with each rents object, so that I can access it in the nested scope.

However, I would suggest to natively build a solution of this problem in ES cc. @Clinton_Gormley

11

@pungent Can you open a Github issue that describes this limitation of function_score in the context of nested queries? Thanks!

12

Hi,

I just rechecked with @mvg who knows nested query much better and he confirms that you cannot access parent fields from a nested context and that you need to do so from the parent context. So if you need access to a nested field as well you currently can try to model your data differently or work around it maybe in a way that I suggested above (maybe that doesn't work in the end though).

13

Will do @ywelsch

14

Yes @cbuescher I modified my mapping now :frowning: