Applying post_filter after top_hits aggregation not working

[Manas_Mohanty]

My input is like this.

[
  {
    "userId": "u1",
    "status": "success",
    "time": 1719543600008
  },
  {
    "userId": "u1",
    "status": "failure",
    "time": 1719543600007
  },
  {
    "userId": "u1",
    "status": "timeout",
    "time": 1719543600006
  },
  {
    "userId": "u2",
    "status": "Failure",
    "time": 1719543600004
  },
  {
    "userId": "u2",
    "status": "Failure",
    "time": 1719543600003
  }
]

I need the users which are not successful in their last attempt.
Explanation: We need to find out last status of each user and discard the users whose last status is "successful"

I have prepared the query like this. Through this I am able to get all users and their last status. After that I want to filter based on status. Can any one help here in fixing this. I have applied post_filter after aggregation, but still its not filtering.
Can any lead help here ?

{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "data.time": {
              "gte": "1719543600000",
              "lte": "1719584179015",
              "format": "epoch_millis"
            }
          }
        },
        {
          "query_string": {
            "query": "data.type:\"user-stats\""
          }
        }
      ]
    }
  },
  "aggs": {
    "group_by_userId": {
      "terms": {
        "field": "data.userId.keyword"
      },
      "aggs": {
        "users_last_status": {
          "top_hits": {
            "size": 1,
            "sort": [
              {
                "data.time": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}

[Carlos_D (Carlos)]

Hi @Manas_Mohanty :

Post-filter is applied to the search results after aggregations are done, so you won't be able to use post-filter for the aggregations themselves.

I think you can achieve what you're looking for with the bucket selector aggregator - you can use a custom script for retaining specific aggregations or not.

Hope that helps!

[Manas_Mohanty]

@Carlos_D Thank you for the response.
Do we have any performance impact when using custom script ?

[Carlos_D (Carlos)]

The script will need to be evaluated for the results aggregated, so there will be a performance impact. It will depend on the number of results you get from the aggregation.