Nested range aggregation misbehaving

Elastic Stack Elasticsearch
1

Hey guys,

I have posted my question on Github and was told it's not a bug. It behaves like a bug to me.

Environment:

The issue is that I've got a nested range aggregation which does not seem to be affected by the filter aggregation that sits in-front of it. I would have considered it an error with the query should it not be the fact that a terms aggregation put on the same level works just fine.

Here are some of the details:

The query has both range and terms aggregations on the same level. Looking at the result one could spot that the range one yields values having status = "scheduled", which should have been filtered already. The terms aggregation seem to work fine.

Could someone please help me understand what's going on here?

2

I have tried to have a terms aggregation in front of the range aggregation. So, in the terms one I could see the current status is "completed" and I have 7 items. However, going in the range aggregation I have ranges that have items with "scheduled" status in.

3

I find it interesting that I can't seem to reproduce the issue with a minimal dataset..

DELETE /bug

PUT /bug
{
  "mappings": {
    "range_agg": {
      "properties": {
        "id_string": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "status": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "user": {
          "type": "nested",
          "properties": {
            "id_string": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "stats": {
          "type": "nested",
          "properties": {
            "id_string": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "talk_duration_percentage": {
              "type": "double"
            }
          }
        }
      }
    }
  },
  "settings": {
    "index": {
      "number_of_shards": "1",
      "number_of_replicas": "0"
    }
  }
  
}

GET /bug/range_agg/_mapping

POST /bug/range_agg
{
  "id_string": "42dc82e6-025f-4000-8383-cb24e9d2c002",
  "status": "scheduled",
  "user": {
    "id_string": "42dc82e6-025f-4000-8383-cb24e9d2c002"
  },
  "stats": {
    "id_string": "42dc82e6-025f-4000-8383-cb24e9d2c002",
    "talk_duration_percentage": 21
  }
}

POST /bug/range_agg
{
  "id_string": "d944f1fd-0caa-4000-b33b-8035365bef14",
  "status": "scheduled",
  "user": {
    "id_string": "d944f1fd-0caa-4000-b33b-8035365bef14"
  },
  "stats": {
    "id_string": "d944f1fd-0caa-4000-b33b-8035365bef14",
    "talk_duration_percentage": 23
  }
}

POST /bug/range_agg
{
  "id_string": "df9e6082-0db0-4000-ac48-42ab819a81fd",
  "status": "scheduled",
  "user": {
    "id_string": "df9e6082-0db0-4000-ac48-42ab819a81fd"
  },
  "stats": {
    "id_string": "df9e6082-0db0-4000-ac48-42ab819a81fd",
    "talk_duration_percentage": 22
  }
}

POST /bug/range_agg
{
  "id_string": "e82e754c-04d6-4000-8ce2-1f4baa023cb7",
  "status": "completed",
  "user": {
    "id_string": "e82e754c-04d6-4000-8ce2-1f4baa023cb7"
  },
  "stats": {
    "id_string": "e82e754c-04d6-4000-8ce2-1f4baa023cb7",
    "talk_duration_percentage": 25
  }
}

POST /bug/range_agg
{
  "id_string": "7000f2b4-06a6-4000-8aae-5ffd78bd6f11",
  "status": "completed",
  "user": {
    "id_string": "7000f2b4-06a6-4000-8aae-5ffd78bd6f11"
  },
  "stats": {
    "id_string": "7000f2b4-06a6-4000-8aae-5ffd78bd6f11",
    "talk_duration_percentage": 27
  }
}

POST /bug/range_agg/_search
{
  "query": {
    "match_all": {}
  },
  "aggs": {
    "status": {
      "terms": {
        "field": "status.keyword",
        "size": 10
      }
    }
  }
}

POST /bug/range_agg/_search
{
  "query": {
    "match_all": {}
  },
  "size": 0,
  "from": 0,
  "aggs": {
    "doc": {
      "filter": {
        "term": {
          "status.keyword": "completed"
        }
      },
      "aggs": {
        "stats": {
          "nested": {
            "path": "stats"
          },
          "aggs": {
            "talk_duration_stats": {
              "range": {
                "field": "stats.talk_duration_percentage",
                "ranges": [
                  {
                    "from": 20,
                    "to": 25
                  },
                  {
                    "from": 25,
                    "to": 30
                  }
                ]
              },
              "aggs": {
                "status": {
                  "reverse_nested": {},
                  "aggs": {
                    "status": {
                      "terms": {
                        "field": "status.keyword",
                        "size": 10
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
4

So I still think this is a case of https://github.com/elastic/elasticsearch/issues/18412

You'll notice in the ticket that the "non-working" example is in the form of:

aggs:
  <filter aggregation>
    <**nested** query>
  aggs:
    <nested aggregation>:
      <other aggregations>

This matches the layout of your aggregation. First you have "activities" filter aggregation which uses a number of nested queries. This is followed by "stats" nested aggregation, etc etc.

This doesn't work because first the filter aggregation finds all root nested documents that match the nested filter (e.g. at least one of the nested objects matches the filter). This collection of matching root nested documents are passed to the following nested aggregation (and sub-aggs). These root documents match the filter -- they had at least one document that matched -- but also have non-matching objects which are also aggregated.

And that's why you see results that didn't match the filter. The initial filter found:

  • "root documents that have a nested object matching the filter"

But what you want is:

  • "nested objects that match the filter"

It's a subtle but important distinction.

In https://github.com/elastic/elasticsearch/issues/18412, the suggested form is instead:

aggs:
  <nested aggregation>:
    <filter aggregation>:
      <**non-nested** query>
    aggs:
      <other aggregations>

In this derivation, we first run a nested aggregation to go to the nested scope. Then we run a filter aggregation with a regular, non-nested filter. The prior nested aggregation has put us in the scope of the nested documents, so we use regular filters/queries and don't need to wrap them in a nested query like the prior example.

This will filter the nested objects (not the root object) to just the criteria we want, then the rest of the aggregations are run on that.

I do agree that it is confusing/unintuitive, but that's just how the nested aggs/queries/documents work (and why https://github.com/elastic/elasticsearch/issues/18412 exists, since we should really clarify the documentation to make this easier).

Hope that helps!

5

Thank you for the prompt explanation!

I have been working on solving the case as well and have narrowed down the issue to the reverse_nested aggregation that goes to the user object.

Weirdly, specifying nested instead of reverse_nested worked. Not sure why that is - I have no user object nested under stats. Is the range aggregation nullifying the current nested level somehow?

Not sure if this makes sense or not, but.. Just to be sure I have changed it to nested, wrapped in a reverse_nested that goes to the root. It works now.

I can't tell what the issue is as I don't seem to be able to re-create it with a minimal set of data. I presume it has to do with the mapping to a certain degree.

We are using a somewhat dated version - 5.6.
Could there have been an issue with reverse_nested in this version?

P.S Filters are gone now. I've moved everything in the query section, not to pollute the aggregations.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.