Moving filter from query to filtered agg slows things at times x3-x30

[haizaar (Zaar Hai)]

Good day,

ES 2.3.5 is here (yeah, old, I know). I have index with several billion documents of metrics data and run filtering query with aggregations on it. Then I tried to move some of the filters into the aggregations through "filtered aggregation" and noticed that speed dropped x10-x30 times!

Here is the slow query and its execution:

{
  "took": 1737,
  "timed_out": false,
  "_shards": {
    "total": 41,
    "successful": 41,
    "failed": 0
  },
  "hits": {
    "total": 84991,
    "max_score": 0,
    "hits": []
  },
}

Here is the fast query and its execution:

{
  "took": 330,
  "timed_out": false,
  "_shards": {
    "total": 41,
    "successful": 41,
    "failed": 0
  },
  "hits": {
    "total": 38607,
    "max_score": 0,
    "hits": []
  }

I understand that the second, fast query, has filter on the query level, and thus points aggregation considers twice as less documents, but still the difference is only about 40k documents and thus difference in the execution time looks strange.

The problem is that this issue is not consistent. Some times they both execute at about the same speed, but at times the slower ones can run up to 5-7 seconds, while, at the same moment, the fast one never goes over 500ms.

Any ideas?

Thanks,
Zaar

P.S. Here is the data model:

{
    "ts": 1502163300,
    "bs": "8k_64k",
    "rw": "r",
    "i": 100,
    "li": 100,
    "lo": 100,
    "t": 100,
    "v": "4",
    "vn": "test",
    "lc": 100,
    "mid": "5218"
  }