Geospatial query with aggregations does not return any results ES7.6

vmanohar1 · July 10, 2020, 8:37pm

1) MY template for indices of type myindex-YYYY-MM-DD is -

PUT _template/myindex_template
{
    "index_patterns" : ["myindex-*"],
    "aliases" : {
      "myindex_alias" : {}
    },
    "settings": {
    "number_of_replicas": 2,
    "number_of_shards": 2,
    "analysis": {},
    "refresh_interval": "1m",
    "index.mapping.ignore_malformed": true
   },
    "mappings": {
            "properties": {
                "my_id"              : {"type" : "long"},
                "my_ts"           : {"type": "date"},
                "location1"               : {"type": "geo_shape"},
                "prev_my_ts"      : {"type": "date"},
                "prev_location1"          : {"type": "geo_shape"}
            }
        }
}  <---- Here my_id,my_ts gives uniqueness(mentioning for understanding purposes.

2) My Query is -->

GET /myindex_alias/_search
{
  "size" : 0,
  "query": {
    "bool": {
      "must": {
        "range": {
          "my_ts": {
            "gte": "2020-06-05T00:00:59",
            "lte": "2020-06-12T21:59:59"
          }
        }
      },
      "filter": {
        "bool": {
          "should": [
            {
              "bool": {
                "must": [
                  {
                    "geo_shape": {
                      "location1": {
                        "indexed_shape": {
                              "index": "my_polygons",
                              "id": "mypoly_1",
                              "path": "polygon_shape"
                        },
                        "relation": "disjoint"
                      }
                    }
                  },
                  {
                    "geo_shape": {
                      "prev_location1": {
                        "indexed_shape": {
                              "index": "my_polygons",
                              "id": "mypoly_1",
                              "path": "polygon_shape"
                        },
                        "relation": "within"
                      }
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "geo_shape": {
                      "location1": {
                        "indexed_shape": {
                              "index": "my_polygons",
                              "id": "mypoly_1",
                              "path": "polygon_shape"
                        },
                        "relation": "within"
                      }
                    }
                  },
                  {
                    "geo_shape": {
                      "prev_location1": {
                        "indexed_shape": {
                              "index": "my_polygons",
                              "id": "mypoly_1",
                              "path": "polygon_shape"
                        },
                        "relation": "disjoint"
                      }
                    }
                  }
                ]
              }
            }
          ]
        }
      }
    }
  },
  "aggs": {
    "dist_id": {
      "terms": {
        "field": "my_id",
        "size": 10000
      }
    }
  }
}

Index my_polygons has predefined polygon - mypoly_1.
The query mentioned in 2) above, used to work fine in a bit earlier versions than 7.6.
I have millions of rows under myindex_alias related indices(each index is daily index) used in the query.

The query is trying to get unique list of my_id for a given time-range for cases where (location1 is inside polygon mypoly_1 and prev_location1 is outside) or the opposite.

Not sure why it does not give any results ( I am certain that it is supposed to give results).

Result is -

{
  "took" : 13,
  "timed_out" : false,
  "_shards" : {
    "total" : 98,
    "successful" : 98,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "dist_id" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [ ]
    }
  }
}

`when it should be giving unique values if my_id (along with count for each)..``

I do not get any errors when I run the query. Is there some setting I am missing here? Has anything changed?? I am certain that same query worked in earlier versions of ES (including I think 7.4).

Please review. Thanks.

Ignacio_Vera · July 11, 2020, 8:36am

I have tried to reproduce the issue and it works for me. This is what I have done:

I started a 7.6.0 cluster. First I create the my_polygons index with a simple polygon that contains point at (0,0):

PUT my_polygons

POST my_polygons/_mapping
{
  "properties":
    {
      "polygon_shape": {
        "type": "geo_shape"
      }
    }
}

POST my_polygons/_doc/mypoly_1
{
    "polygon_shape": "POLYGON((-1 -1, 1 -1, 1 1, -1 1, -1 -1))"
}

Second, a create the template:

PUT _template/myindex_template
{
    "index_patterns" : ["myindex-*"],
    "aliases" : {
      "myindex_alias" : {}
    },
    "settings": {
    "number_of_replicas": 2,
    "number_of_shards": 2,
    "analysis": {},
    "refresh_interval": "1m",
    "index.mapping.ignore_malformed": true
   },
    "mappings": {
            "properties": {
                "my_id"              : {"type" : "long"},
                "my_ts"           : {"type": "date"},
                "location1"               : {"type": "geo_shape"},
                "prev_my_ts"      : {"type": "date"},
                "prev_location1"          : {"type": "geo_shape"}
            }
        }
}

Then I create an index that matches the template with three documents:

# should match
POST myindex-test/_doc
{
"my_id" : 1,
"my_ts" : "2020-07-02",
"location1" : "POINT(0 0)",
"prev_my_ts" : "2020-07-01",
"prev_location1" : "POINT(100 0)"
}

# should match
POST myindex-test/_doc
{
"my_id" : 2,
"my_ts" : "2020-07-02",
"location1" : "POINT(100 0)",
"prev_my_ts" : "2020-07-01",
"prev_location1" : "POINT(0 0)"
}

# should not match
POST myindex-test/_doc
{
"my_id" : 3,
"my_ts" : "2020-07-02",
"location1" : "POINT(100 0)",
"prev_my_ts" : "2020-07-01",
"prev_location1" : "POINT(50 0)"
}

POST myindex-test/_refresh

Execute the query:


GET /myindex_alias/_search
{
  "size" : 0,
  "query": {
    "bool": {
      "must": {
        "range": {
          "my_ts": {
            "gte": "2020-06-05T00:00:59",
            "lte": "2020-07-12T21:59:59"
          }
        }
      },
      "filter": {
        "bool": {
          "should": [
            {
              "bool": {
                "must": [
                  {
                    "geo_shape": {
                      "location1": {
                        "indexed_shape": {
                              "index": "my_polygons",
                              "id": "mypoly_1",
                              "path": "polygon_shape"
                        },
                        "relation": "disjoint"
                      }
                    }
                  },
                  {
                    "geo_shape": {
                      "prev_location1": {
                        "indexed_shape": {
                              "index": "my_polygons",
                              "id": "mypoly_1",
                              "path": "polygon_shape"
                        },
                        "relation": "within"
                      }
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "geo_shape": {
                      "location1": {
                        "indexed_shape": {
                              "index": "my_polygons",
                              "id": "mypoly_1",
                              "path": "polygon_shape"
                        },
                        "relation": "within"
                      }
                    }
                  },
                  {
                    "geo_shape": {
                      "prev_location1": {
                        "indexed_shape": {
                              "index": "my_polygons",
                              "id": "mypoly_1",
                              "path": "polygon_shape"
                        },
                        "relation": "disjoint"
                      }
                    }
                  }
                ]
              }
            }
          ]
        }
      }
    }
  },
  "aggs": {
    "dist_id": {
      "terms": {
        "field": "my_id",
        "size": 10000
      }
    }
  }

I get the expected results:

{
  "took" : 23,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "dist_id" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : 1,
          "doc_count" : 1
        },
        {
          "key" : 2,
          "doc_count" : 1
        }
      ]
    }
  }
}

Could you provide an example that reproduces your issue?

vmanohar1 · July 12, 2020, 11:54am

It can be data issue itself.. Let me verify that for a few days..
I had tried with fake small data and it works.. Thanks for replying. I will update in a few days.

vmanohar1 · July 13, 2020, 3:06pm

It seems to be data issue!! We are getting corrected data from the stream again..
Thank you for reviewing this ticket.

vmanohar1 · July 14, 2020, 6:27pm

Verified that all works after correction in data feed.. Ticket can be closed. Thanks.

system · August 11, 2020, 6:27pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.