How to find a term (title: req.query['q]) with geo_distance with elasticsearch 8.6

teoman_kirac · February 12, 2023, 10:20am

Years ago I had this working with elasticsearch 16.x.x. It looked like this:

  let body = {
    size: 200,
    from: 0,
    query: {
      bool: {
        must: {
          term: { title : req.query['q'] }
        },
        filter: {
          geo_distance: {
            distance: req.query['distance'],
            location: {
              lat: req.query['lat'],
              lon: req.query['lon']
            }
          }
        }
      }
    }
  }

Now (after upgrading to 8.6) it plum doesn't work. Can some one help, please. It is so strange looking at 15 lines of code and honestly going over every variation imaginable to not find a solution. Here is my latest attempt:

    let body = {
      "query": {
        "bool": {
          "must": {
            "term": { title : req.query['q'] }
          }
        }
      },
      "sort": [
        {
          "_geo_distance": {
            "distance": req.query['q'],
            "coords": {
              "lat": req.query['lat'],
              "lon": req.query['lon']
            },
            "order": "asc",
            "unit": "km",
            "mode": "min",
            "distance_type": "arc",
            "ignore_unmapped": true
          }
        }
      ]
    }

the error log is complaining about location not being a field, and other odds and ends like; Caused by:
illegal_argument_exception: unsupported symbol [i] in geohash [si]
Root causes:
parse_exception: unsupported symbol [i] in geohash [si]

Should I use filter? or Sort? Or?

Thank you for reading!

Christian_Dahlqvist · February 12, 2023, 10:30am

What is the mapping of the fields being queried?

This seem to not be consistent between the queries. Is this an error?

teoman_kirac · February 12, 2023, 10:43am

Hey Christian! Thanks for the response

mappings: {
location: {
type: "geo_point"
}

and from the back as;


{
  "posts": {
    "mappings": {
      "properties": {
      "location": {
          "properties": {
            "_latitude": {
              "type": "float"
            },
            "_longitude": {
              "type": "float"
            }
          }
....

teoman_kirac · February 12, 2023, 10:58am

and I think I wasn't using quotes when it was working. in regards to:

This seem to not be consistent between the queries. Is this an error?

Christian_Dahlqvist · February 12, 2023, 10:59am

That does not be mapped as a geo point. Have a look at the docs for an example of what mappings and query should look like.

teoman_kirac · February 12, 2023, 12:18pm

i still don't understand.

from a firestore collection i get /posts/_mappings:

{
  "posts": {
    "mappings": {
      "properties": {
      "location": {
          "properties": {
            "_latitude": {
              "type": "float"
            },
            "_longitude": {
              "type": "float"
            }
          }
      }
    }
}

this below created that above

mappings: {
      location: {
        type: "geo_point"
      }
    },

I have a ts array ref:


import { Reference } from "./types"
const references: Array<Reference> = [
  {
    collection: "posts",
    index: "posts",
    // type: "posts", THIS USED TO DO SOMETHING AND IS NOT ALLOWED NOW. NOT SURE WHAT
    mappings: {
      location: {
        type: "geo_point"
      }
    },

    transform: (data, parent) => ({
      ...data,
    })
  }
]
export default references

that does the below: (import { Reference } from "./types")

import * as admin from "firebase-admin"
import { Client } from "@elastic/elasticsearch"
import { CollectionReference } from "@google-cloud/firestore"

export type ElasticSearchFieldType =
  | "text"
  | "keyword"
  | "date"
  | "long"
  | "double"
  | "boolean"
  | "ip"
  | "geo_point"
  | "geo_shape"
  | "completion"

export type FirebaseDocChangeType = "added" | "modified" | "removed"

export type DynamicTypeIndex = (
  snap?: admin.firestore.DocumentSnapshot,
  parentSnap?: admin.firestore.DocumentSnapshot
) => string

export interface ElasticSearchOptions {
  requestTimeout: number
}

export interface Reference {
  collection: string
  subcollection?: string
  index: DynamicTypeIndex | string
  include?: Array<string>
  exclude?: Array<string>
  mappings?: { [key: string]: { type: ElasticSearchFieldType; format?: string; dynamic?: boolean } }
  builder?: (ref: CollectionReference) => admin.firestore.Query
  subBuilder?: (ref: CollectionReference) => admin.firestore.Query
  filter?: (data: admin.firestore.DocumentData) => boolean | null
  transform?: (data: { [key: string]: any }, parentSnap?: admin.firestore.DocumentSnapshot) => { [key: string]: any }
  onItemUpserted?: (
    data: { [key: string]: any },
    parentSnap: admin.firestore.DocumentSnapshot,
    client: Client
  ) => void | Promise<void>
}

I don't understand. argh. Sorry to just dump code here, but mappings from firestore are fixed. And now I need to query that data into a format that is understood by client, correct?

Do I need to nest and double nest properties and pin fields in the mappings in my ref?

Thank you for your time.

teoman_kirac · February 13, 2023, 8:18am

like what does this mean:

Root causes:
	parsing_exception: Trying to reset fieldName to [location], already set to [distance].

regarding this:

sort: [
        {
          _geo_distance: {
            distance: req.query['distance'],
            distance_type: "arc",
            location: {
              lat: req.query['lat'],
              lon: req.query['lon']
            },
            order: "asc",
            unit: "km",
            mode: "min",
            // distance_type: "arc",
            ignore_unmapped: true
          }
        }
      ],
      query: {
        bool: {
          must: {
            term: { title : req.query['q'] }
          }
        }
      }

Ignacio_Vera · February 13, 2023, 8:59am

The issue here is that geo_distance sort does not support a distance field and it is treating it as the field containing the point to be sorted. Then there are two errors possible:

It cannot parse the content of the field as a point, then you get errors like the one above:

illegal_argument_exception: unsupported symbol [i] in geohash [si]
Root causes:
parse_exception: unsupported symbol [i] in geohash [si]

If it parses the content on distance, then it fails when finding the location field as it has already set the coordinate value:

Root causes:
	parsing_exception: Trying to reset fieldName to [location], already set to [distance].

To fix it, please remove the distance field as it is not needed for sorting. Feel free to open an issue to improve the error messages as I think we should provide better guidance here.

teoman_kirac · February 13, 2023, 9:16am

Thanks Ignacio, will open a new subject for that.

But then regarding a reference point to calculate distance is achieved how?

When I use a filter rather than sort to return the objects within a distance, I get that error as well.

I need to return all posts within a specified distance (distance: req.query['distance']) to lat lon.

How can this be achieved. I had this working a while ago. Now it does not, which is half of my hang up right now.

Ignacio_Vera · February 13, 2023, 9:46am

A geo_distance filter should work. It is not clear to me from the post above which is the geo_distance filter that is failing and what is the error.

Could you post the exact geo_distance filter in 8.6 you are building and the exact error?

teoman_kirac · February 13, 2023, 9:55am

Sorry all over the place. I have tried quite a few variations but this is the one that used to work:


    let body = {
      size: 200,
      from: 0,
      query: {
        bool: {
          must: {
            term: { title : req.query['q'] }
          },
          filter: {
            geo_distance: {
              distance: req.query['distance'],
              location: {
                lat: req.query['lat'],
                lon: req.query['lon']
              }
            }
          }
        }
      }
    }

This gets:

	Root causes:
		query_shard_exception: failed to find geo field [location]

  Root causes:
  query_shard_exception: failed to find geo field [location]
  at SniffingTransport.request (node_modules/@elastic/transport/lib/Transport.js:479:27)
  at processTicksAndRejections (node:internal/process/task_queues:96:5)
  at async Client.SearchApi [as search] (node_modules/@elastic/elasticsearch/lib/api/api/search.js:66:12)

teoman_kirac · February 13, 2023, 9:56am

when i remove filter, I get the desired results on the term..

 let body = {
      size: 200,
      from: 0,
      query: {
        bool: {
          must: {
            term: { title : req.query['q'] }
          }
          // filter: {
          //   geo_distance: {
          //     distance: req.query['distance'],
          //     location: {
          //       lat: req.query['lat'],
          //       lon: req.query['lon']
          //     }
          //   }
          // }
        }
      }
    }

results {                                                                                                                                                                                                         11:58:25
  took: 6,
  timed_out: false,
  _shards: {
    total: 1,
    successful: 1,
    skipped: 0,
    failed: 0
  },
  hits: {
    total: {
      value: 1,
      relation: 'eq'
    },
    max_score: 1.6375021,
    hits: [
      [Object]
    ]
  }
}

Ignacio_Vera · February 13, 2023, 10:24am

The error is clear on what is going on, your location field is not a geo field, something is not working as expected between your mappings and the data you are ingesting. You said that from the mappings API you see the following:

{
  "posts": {
    "mappings": {
      "properties": {
      "location": {
          "properties": {
            "_latitude": {
              "type": "float"
            },
            "_longitude": {
              "type": "float"
            }
          }
      }
    }
}

location is being mapped as two floats, something is failing there and the value is not mapped to a geo_point. It is very strange the _latitude and _longitude names, that seems wrong. Could you share the format of one of the geo_point you are ingesting?

teoman_kirac · February 13, 2023, 10:39am

I thought that was the desired format. But good to know!

When i do this:

    // Transform data as we receive it from firestore
    transform: (data, parent) => ({
      ...data,
      // location: new GeoPoint(data.location._latitude,data.location._longitude)
      // latitude: `${data.location._latitude}`,
      // longitude: `${data.location._longitude}`
      location:`${data.location._latitude},${data.location._longitude}`
    })

i get: (with GET posts/_mappings.


        "location": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },

and this error:

query_shard_exception: Field [location] is of unsupported type [text] for [geo_distance] query

  Root causes:
  query_shard_exception: Field [location] is of unsupported type [text] for [geo_distance] query
  at SniffingTransport.request (node_modules/@elastic/transport/lib/Transport.js:479:27)
  at processTicksAndRejections (node:internal/process/task_queues:96:5)
  at async Client.SearchApi [as search] (node_modules/@elastic/elasticsearch/lib/api/api/search.js:66:12)

teoman_kirac · February 13, 2023, 10:41am

I tried all of those variations in the transform()

teoman_kirac · February 13, 2023, 10:45am

This is what I get in console with filter commented out, and this as well ( //location:${data.location._latitude},${data.location._longitude}) commented out in transform:

location: "32.04935700000001,34.75835500000001"

and back to the way I had it originally returns this:

1. location: Object
  1. _latitude: 32.04935700000001
  2. _longitude: 34.75835500000001

I can't create a "location" definition in transform that gets me the format i need. hmmm.

Ignacio_Vera · February 13, 2023, 11:24am

Thanks!

I am not familiar with the way you are doing things but I would expect this to work:

// Transform data as we receive it from firestore
    transform: (data, parent) => ({
      ...data,
      // location: new GeoPoint(data.location._latitude,data.location._longitude)
      // latitude: `${data.location._latitude}`,
      // longitude: `${data.location._longitude}`
      location:`${data.location._latitude},${data.location._longitude}`
    })

If you have created the mapping or an index template before ingesting data . Are you using an explicit mapping or an index template? Has this been execute before indexing data?

teoman_kirac · February 13, 2023, 1:02pm

Thanks Ignacio, I am using elasticstore (GitHub - acupofjose/elasticstore: A pluggable union between Firebase CloudFirestore + ElasticSearch). Took some time to get it to work with elastic 8.6, and other dependencies. Will dive into it to see if it's something to do with that.

Ignacio_Vera · February 13, 2023, 1:13pm

mmh, I see that repository has not been updated in a while and predates the release of Elasticsearch 8.0.0. More over in the packages it seems to have been tested with Elasticsearch 7.8.0:

 "@elastic/elasticsearch": "^7.8.0",

I wonder if there is some incompatibility with 8.0, in particular due to the removal of types: Removal of mapping types | Elasticsearch Guide [8.6] | Elastic

system · February 13, 2023, 1:13pm

Elasticsearch 7.8 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )