# Issue after migration from geo\_shape pre-fix based to geo\_shape based bkd tree mapping

**URL:** <https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [August 4, 2022, 11:15am UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406 "2022-08-04T11:15:22Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abhilash\_Venkatesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_venkatesh/32/100717_2.png) [@Abhilash\_Venkatesh](https://discuss.elastic.co/u/Abhilash_Venkatesh)\
**Post date:** [August 4, 2022, 11:15am UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406/1 "2022-08-04T11:15:22Z")

</div>

Hi

**Problem Description**  
We want to migrate from elastic 7.x to 8. Migrated to 7.17.5 and resolved issues.  
One of it was was location field with geo\_shape mapping, we used `strategy : recursive` . To resolve took backup (backup-of-index-surat, this too has explicit mapping , geo\_shape with `strategy : recursive`) and reindexed to the index with explicit mapping (dynamic: false) containing location with just geo\_shape mapping without any strategy params. All documents were reindexed (counts matched)

The number of documents returned for geo\_shape (circle, bbox, polygon) queries on older mapping and newer mapping for same data are different. Following are two queries :

1. Linestring query: This w.r.t our data returns around 40K docs on older mapping and in newer, it returns zero docs

```auto
GET backup-of-index-surat/_count
{
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "id": [
              "iisc.ac.in/89a36273d77dac4cf38114fca1bbe64392547f86/rs.iudx.io/surat-itms-realtime-information/surat-itms-live-eta"
            ],
            "boost": 1
          }
        },
        {
          "geo_shape": {
            "location": {
              "shape": {
                "type": "linestring",
                "coordinates": [
                  [
                    72.842,
                    21.2
                  ],
                  [
                    72.923,
                    20.8
                  ]
                ]
              },
              "relation": "intersects"
            }
          }
        },
        {
          "range": {
            "observationDateTime": {
              "from": "2020-10-12T00:00:00.000Z",
              "to": "2020-10-22T00:00:00Z",
              "include_lower": true,
              "include_upper": true,
              "boost": 1
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  }
}

```

1. geo\_shape circle query: This returns around 4k docs in older mapping and only 2k in newer mapped index

```auto
GET backup-of-index-surat/_count
{
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "id": [
              "iisc.ac.in/89a36273d77dac4cf38114fca1bbe64392547f86/rs.iudx.io/surat-itms-realtime-information/surat-itms-live-eta"
            ],
            "boost": 1
          }
        },
        {
          "geo_shape": {
            "location": {
              "shape": {
                "radius": "10.0m",
                "type": "Circle",
                "coordinates": [
                  72.834,
                  21.178
                ]
              },
              "relation": "within"
            }
          }
        },
        {
          "range": {
            "observationDateTime": {
              "from": "2020-10-12T00:00:00.000Z",
              "to": "2020-10-22T00:00:00Z",
              "include_lower": true,
              "include_upper": true,
              "boost": 1
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  }
}

```

**What do we expect/Questions**

- Is this reindexing from pre-fix based geo\_shape to bkd based geo\_shape, the way to migrate the index to 8 version compatible?

- why does the same geo\_shape query return different number of docs for same data drastically , with just change on how its stored in elastic? The precision guaranteed by bkd tree (7 floating point) is higher than the data values stored (6 floating point) ? Is there need to change query to work with newer mapping?

- The geo\_shape queries should return same number of docs after migrating to BKD tree based representation.

**Some relevant details:**  
Elastic version : 7.17.5 running as docker container  
Data: All docs in the above indexed dataset are of Point type geo\_shape location data. The latitude, longitude data values are of precision 6 floating points.

Please let me know if anything else is needed.

---

<div class="post-metadata">

**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\
**Post date:** [August 4, 2022, 4:18pm UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406/2 "2022-08-04T16:18:22Z")

</div>

Hi,

I would recommend to read the following blog, in particular the section why we needed a new approach. After that if you have more questions I will be happy to try to answer them:

> **[BKD-backed geo\_shapes in Elasticsearch: precision + efficiency + speed](https://www.elastic.co/blog/bkd-backed-geo-shapes-in-elasticsearch-precision-efficiency-speed)**
>
> The new default indexing technique for geo\_shape field types in the Elasticsearch 7.0+ delivers an immense performance improvement over the legacy prefix tree indexing approach for geospatial geometries. Learn more about how geo\_shapes have evolved...

Thanks!

---

<div class="post-metadata">

**Author:** ![Abhilash\_Venkatesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_venkatesh/32/100717_2.png) [@Abhilash\_Venkatesh](https://discuss.elastic.co/u/Abhilash_Venkatesh)\
**Post date:** [August 4, 2022, 5:56pm UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406/3 "2022-08-04T17:56:24Z")

</div>

Thank you @Ignacio_Vera . Went through the post. This answers my first question and would want to migrate to have better performance and size.

So coming back to the core of the issue we faced. We migrated the existing index with geo\_shape based on prefix-tree for the location data to BKD-tree based geo\_shape mapping using reindex API. This resulted in geo\_shape queries on the new index returning a lesser number of docs ( or zero docs in case of linestring query) than what was from the older mapped index. This sort of looks like a bug and not able to understand why geo\_shape queries on the two indexes return different counts.

The first post gives examples of two geo\_shape queries.

The latitude, and longitude data values are 1e-6 precision, and BKD is capable to store more precisely. The mismatch between search documents returned, should not be because of precision.

---

<div class="post-metadata">

**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\
**Post date:** [August 5, 2022, 10:49am UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406/4 "2022-08-05T10:49:41Z")

</div>

> [@Abhilash\_Venkatesh](#):
>
> Data: All docs in the above indexed dataset are of Point type geo\_shape location data. The latitude, longitude data values are of precision 6 floating points.

All your data are just points? I would recommend to use geo\_point field type instead of geo\_shape in that case as it is faster and it will use less space.

> [@Abhilash\_Venkatesh](#):
>
> This resulted in geo\_shape queries on the new index returning a lesser number of docs ( or zero docs in case of linestring query) than what was from the older mapped index. This sort of looks like a bug and not able to understand why geo\_shape queries on the two indexes return different counts.

This is expected as the precision on the old strategy depends on the precision of the grid and sometimes even in the area of the geometry. The new indexing system has a fix error due to the encoding of latitudes / and longitudes.

Having said that, querying points with line strings is tricky, what is the use case? you will normally face precision errors just because of floating point arithmetic.

---

<div class="post-metadata">

**Author:** ![Abhilash\_Venkatesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_venkatesh/32/100717_2.png) [@Abhilash\_Venkatesh](https://discuss.elastic.co/u/Abhilash_Venkatesh)\
**Post date:** [August 5, 2022, 3:57pm UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406/5 "2022-08-05T15:57:34Z")

</div>

> All your data are just points? I would recommend to use geo\_point field type instead of geo\_shape in that case as it is faster and it will use less space.

yes, its just points and basically its bus live location data. Thanks you for suggestion, will use geo\_point.

A side question: Is there an elastic query to filter points type data, polygon type data?

> This is expected as the precision on the old strategy depends on the precision of the grid and sometimes even in the area of the geometry. The new indexing system has a fix error due to the encoding of latitudes / and longitudes.

This make sense. The indexed data in older strategy was less precise ([with default 50 m precision](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/geo-shape.html)). So the queries on this older strategy index gave more count for all geo\_shape queries - circle,bbox, and most probably also the reason it gave certain docs for linestring in older mapping.

> Having said that, querying points with line strings is tricky, what is the use case? you will normally face precision errors just because of floating point arithmetic.

use case for linestring - to find buses which travel along the same route/path/sub-path.

---

<div class="post-metadata">

**Author:** ![Abhilash\_Venkatesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_venkatesh/32/100717_2.png) [@Abhilash\_Venkatesh](https://discuss.elastic.co/u/Abhilash_Venkatesh)\
**Post date:** [August 8, 2022, 9:34am UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406/6 "2022-08-08T09:34:58Z")

</div>

> Having said that, querying points with line strings is tricky, what is the use case?

If you can elaborate a bit more on why its tricky?

---

<div class="post-metadata">

**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\
**Post date:** [August 8, 2022, 10:06am UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406/7 "2022-08-08T10:06:57Z")

</div>

In the new indexing strategy a point will only intersects line if it exactly lies on the line which is sensible to double floating point errors. What you really want is to add a buffer on your lines so you can match any point at a distance of a line.

Unfortunately we are not (yet) providing that functionality so you might need to buffer the line at the client, for example transforming the line into a polygon and use the polygon to query your index.

Hope it makes sense.

---

<div class="post-metadata">

**Author:** ![Abhilash\_Venkatesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_venkatesh/32/100717_2.png) [@Abhilash\_Venkatesh](https://discuss.elastic.co/u/Abhilash_Venkatesh)\
**Post date:** [August 9, 2022, 3:13pm UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406/8 "2022-08-09T15:13:48Z")

</div>

> [@Ignacio\_Vera](#):
>
> In the new indexing strategy a point will only intersects line if it exactly lies on the line which is sensible to double floating point errors

Thanks for the explanation. Did a quick check, and even giving the same two points present in index as coordinates for linestring query, the docs did not match. My understanding, while storing of index, the precision is reduced to 10-7 and while searching it still does with greater precision - double floating point.

Doesn't having both work at same precision levels makes more sense?

> [@Ignacio\_Vera](#):
>
> What you really want is to add a buffer on your lines so you can match any point at a distance of a line

This feature would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\
**Post date:** [August 9, 2022, 3:25pm UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406/9 "2022-08-09T15:25:50Z")

</div>

> Doesn't having both work at same precision levels makes more sense?

You need to be in Elasticsearch 8.3+ for that: [https://github.com/elastic/elasticsearch/pull/85441](https://github.com/elastic/elasticsearch/pull/85441)

> This feature would be greatly appreciated.

I agree and it is in the roadmap. There seems to be no issue yet, will try to create one soon.

---

<div class="post-metadata">

**Author:** ![Abhilash\_Venkatesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_venkatesh/32/100717_2.png) [@Abhilash\_Venkatesh](https://discuss.elastic.co/u/Abhilash_Venkatesh)\
**Post date:** [August 12, 2022, 3:54am UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406/10 "2022-08-12T03:54:46Z")

</div>

> [@Ignacio\_Vera](#):
>
> You need to be in Elasticsearch 8.3+ for that: [Quantize geo queries to remove true negatives from search results by iverase · Pull Request #85441 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/85441)

After updating the elasticsearch to 8.3.3, the linestring query between two points returned documents.

Thank you for all the support!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 9, 2022, 3:54am UTC](https://discuss.elastic.co/t/issue-after-migration-from-geo-shape-pre-fix-based-to-geo-shape-based-bkd-tree-mapping/311406/11 "2022-09-09T03:54:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.