Very slow aggregation performance for trivial aggs

Elastic Stack Elasticsearch
2

ts field is indexed ... finding minimal value for it is a matter of O(1) operation on each segment of every shard

What you are describing is theoretically the best speed attainable if you ignore everything else aggregations are designed to do.
Let's look at what is missing from your example:

  1. No query to subset the data (e.g. "find all records with IP address X")
  2. No parent aggregation (e.g. a terms agg to group by "machine" field value)
  3. No filtering (e.g. an alias restricting access to docs via a filter)

Now consider that the max/min etc "leaf" aggregations are designed to sit at the end of this chain of filters and look up values for each doc produced by this stream. There's no cheating or shortcuts in these implementations that look in the inverted index for the last value in the global terms enum - they always expect to be processing a stream of (potentially filtered) doc IDs and each value for each doc needs to be retrieved to determine the max. So your cost will be O(n) where n is the number of docs on a shard. Obviously when you are using DocValues OS-level file system caching is going to be a big help.

1 Like