# Using post filter to remove entries whereby collapsed inner hits total is less than X

**URL:** <https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722>\
**Category:** Elasticsearch\
**Created:** [December 9, 2024, 6:54pm UTC](https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722 "2024-12-09T18:54:43Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![codeMonkey82](https://avatars.discourse-cdn.com/v4/letter/c/96bed5/32.png) [@codeMonkey82](https://discuss.elastic.co/u/codeMonkey82)\
**Post date:** [December 9, 2024, 6:54pm UTC](https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722/1 "2024-12-09T18:54:43Z")

</div>

I have a query that collapses on a field representing a hash that can at most be shared between two entries. What I need to do is via a post filter (or alternative) remove the results from the final list whereby the inner hits total is 1 and not 2, however post filter can not find the inner hits for each entry and hence the total is not available.

What would be the best way to filter results whereby they only have one entry within the collapse inner hits ?

Elastic version : 7.10

Example query below...

```auto
{
  "query": {
    "bool": {
      "must": [
        {
          "bool": {
            "must": [
              {
                "exists": {
                  "field": "someHash"
                }
              }
            ]
          }
        },
        {
          // ... other must items ... //
        }
      ],
      "filter": [
        // ... filter items ... //
      ],
      "must_not": [
        // ... must not items ... //
      ]
    }
  },
  "collapse": {
    "field": "someHash",
    "inner_hits": {
      "name": "same_hash",
      "size": 2
    }
  },
  "aggs": {
    "unique_count": {
      "cardinality": {
        "field": "someHash",
        "precision_threshold": 10000
      }
    }
  },
  "sort": [
    {
      "entityId": "desc"
    }
  ],
  "post_filter": {
    "bool": {
      "filter": [
        {
          "term": {
            "inner_hits.same_hash.hits.total": {
              "value": 2
            }
          }
        }
      ]
    }
  },
  "from": 0,
  "size": 100
}

```

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [December 10, 2024, 8:40am UTC](https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722/2 "2024-12-10T08:40:25Z")

</div>

Hi @codeMonkey82,

Thanks for raising your issue. It looks like it might be similar to the below issue:

> [@Applying post\_filter to limit inner\_hits in nested document](https://discuss.elastic.co/t/applying-post-filter-to-limit-inner-hits-in-nested-document/207925):
>
> Hi, I'm struggling how to apply a post\_filter to some nested documents. What I'd like to do is have some aggregation on all my nested documents, but have only certain nested documents returned (the general idea of a post\_filter). I'm succesfull in applying the post\_filter to filter out the root (parent)-documents, but not in filtering the inner hits on this document. As an example, I've created two documents that each contain 2 people (with a name and age), and what I'd like to have returned i…

Can you give the nested/ inner\_hits option a try and let us know if that resolves your issue?

---

<div class="post-metadata">

**Author:** ![codeMonkey82](https://avatars.discourse-cdn.com/v4/letter/c/96bed5/32.png) [@codeMonkey82](https://discuss.elastic.co/u/codeMonkey82)\
**Post date:** [December 10, 2024, 11:13am UTC](https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722/3 "2024-12-10T11:13:00Z")

</div>

@carly.richmond Unfortunately I did try this but it did not work...

I tried the following query...

```auto
{
  "query": {
    "bool": {
      "must": [
        {
          "bool": {
            "must": [
              {
                "exists": {
                  "field": "someHash"
                }
              }
            ]
          }
        },
        {
          // ... other must items ... //
        }
      ],
      "filter": [
        // ... filter items ... //
      ],
      "must_not": [
        // ... must not items ... //
      ]
    }
  },
  "collapse": {
    "field": "someHash",
    "inner_hits": {
      "name": "same_hash",
      "size": 2
    }
  },
  "aggs": {
    "unique_count": {
      "cardinality": {
        "field": "someHash",
        "precision_threshold": 10000
      }
    }
  },
  "sort": [
    {
      "entityId": "desc"
    }
  ],
  "post_filter": {
    "nested": {
      "path": "same_hash",
      "inner_hits": {},
      "query": {
        "term": {
          "hits.total": {
            "value": 2
          }
        }
      }
    }
  },
  "from": 0,
  "size": 100
}

```

The error response was...

failed to create query: [nested] failed to find nested object under path [same\_hash]

FYI it is not the contents of the inner hits I am interested in but rather the total which will allow me to filter out the parent for less than two inner hits

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [December 10, 2024, 12:32pm UTC](https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722/4 "2024-12-10T12:32:17Z")

</div>

Thanks for confirming. Can you share the mapping for your index please? Also which version of Elasticsearch are you using?

---

<div class="post-metadata">

**Author:** ![codeMonkey82](https://avatars.discourse-cdn.com/v4/letter/c/96bed5/32.png) [@codeMonkey82](https://discuss.elastic.co/u/codeMonkey82)\
**Post date:** [December 10, 2024, 12:46pm UTC](https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722/5 "2024-12-10T12:46:03Z")

</div>

So I pulled one of the mappings but had to strip out certain elements for security reasons but fundamentally the contents is wrapped the same. I am using Elasticsearch v7.10.

```auto
{
  "index_2024_06": {
      "mappings": {
          "dynamic": "false",
          "properties": {
              "hasTask": {
                  "type": "boolean"
              },
              "someHash": {
                  "type": "keyword"
              },
              "entityId": {
                  "type": "keyword"
              }
          }
      }
  }
}

```

---

<div class="post-metadata">

**Author:** ![codeMonkey82](https://avatars.discourse-cdn.com/v4/letter/c/96bed5/32.png) [@codeMonkey82](https://discuss.elastic.co/u/codeMonkey82)\
**Post date:** [December 11, 2024, 10:26am UTC](https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722/6 "2024-12-11T10:26:12Z")

</div>

@carly.richmond any joy on this one, really need a solution that works or we might need to refactor our approach around the use of collapse.

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [December 13, 2024, 9:06am UTC](https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722/7 "2024-12-13T09:06:54Z")

</div>

Sorry I've not had time to have a play yet to figure it out. It looks like the `post_filter` approach in the referenced post is available from 7.10.

I see the type of `someHash` is `keyword` rather than `nested` which surprised me slightly. Can you give me an example of the data structure (sanitized obviously!)?

---

<div class="post-metadata">

**Author:** ![codeMonkey82](https://avatars.discourse-cdn.com/v4/letter/c/96bed5/32.png) [@codeMonkey82](https://discuss.elastic.co/u/codeMonkey82)\
**Post date:** [December 13, 2024, 9:34am UTC](https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722/8 "2024-12-13T09:34:51Z")

</div>

Nested is only used for arrays to allow them to be searched independently but what I am trying to do is query the inner\_hits total value which happens as part of the collapse. If I were able to query this total (inner\_hits.same\_hash.hits.total.value) then I would be able to filter out what I need but I am unable to do this. It seems that inner\_hits is not available to the post\_filter in this way. So I need a solution to the above.

I suppose the question first is does the post filter run against the results post collapse ?

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [December 13, 2024, 12:21pm UTC](https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722/9 "2024-12-13T12:21:21Z")

</div>

Thanks for confirming @codeMonkey82 . I've not been able to get the `post_filter` to work either. But I was wondering if you had considered using a `terms` aggregation combined with a `bucket_selector` to filter the counts instead of using collapse?

> **[Bucket selector aggregation | Elasticsearch Guide \[7.10\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/search-aggregations-pipeline-bucket-selector-aggregation.html)**

---

<div class="post-metadata">

**Author:** ![codeMonkey82](https://avatars.discourse-cdn.com/v4/letter/c/96bed5/32.png) [@codeMonkey82](https://discuss.elastic.co/u/codeMonkey82)\
**Post date:** [December 13, 2024, 12:24pm UTC](https://discuss.elastic.co/t/using-post-filter-to-remove-entries-whereby-collapsed-inner-hits-total-is-less-than-x/371722/10 "2024-12-13T12:24:24Z")

</div>

The solution needs to take into pagination and sorting of the final result set so using buckets might not work as expected, I do appreciate the response thought so thank you