Demystifying Performance- Fast Queries, Slow Scans?

Elastic Stack Elasticsearch
1

I've noticed I can run fairly complex, dynamic count queries and return an exact count over millions of documents. I would assume if searching is so quick, then the only performance bottlenecks for a scan to return the results would be in the network itself.

However, i'm finding scans are extremely cpu intensive operations. I can see from a query that my search applies to ~5mm documents in sub second time (with no noticeable cpu hit), then when I start pulling the data back through the scrolling api, the cpu hits 100% until the scroll is finished.

Could I get some insights into what Elasticsearch does when a scan is happening? Why are queries so easy to perform where scans are very difficult? If I can determine the bottleneck, I can make changes to mitigate those bottlenecks, but I can't make sense/pinpoint where the cpu would be struggling.

2

Aggregations are run based on indexes and generally do not require access to the document source. When you retrieve documents, e.g. through a scroll query, Elasticsearch need to fetch the indexed documents which has the potential to require much more disk I/O and therefore be slower.

3

That must be it. I just wouldn't expect so much CPU work for I/O.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.