Performance issues around _source and large page size

Elastic Stack Elasticsearch
1

We're running a ES 1.7 cluster with an index of ~1.5 million documents, averaging around 20k in size. Our searches generally use a page size of 1000, as we do secondary sorting with live data that isn't available in our ES Index.

We currently have _source disabled for performance reasons. We're interested in turning it on, but we're seeing a significant performance hit from our current field-based approach.

I created mirrored indices, one with _source enabled, one disabled. Both include stored fields, as we would want those for a transitional period.

For one of our sample queries (filtered, heavily faceted), we see response times like:

Index _source disabled
Query returns Fields
120ms response time

Index _source enabled
Query returns Fields
430ms response time

Index _source enabled
Query returns Filtered Source
750ms response time

Even when we don't query for the _source, response times tripled just because there was _source stored in the index.

Beyond that, when we do return filtered _source, rather than stored fields, response times almost double again.

Has anyone else seen issues like this? Any recommendations for mitigating the perf hit?

2

Way back when, I did observe similar behavior, but didn't do formal tests.
Would be curious to see if you observe same issue on 2.x code base.

3

Did you see a performance improvement when moving to 2.x? We're hoping to make the move before terribly long, but I haven't seen a whole lot of info on performance comparisons between the major versions.

4

Your mileage may vary is the answer. When comparing apples to apples, yes.
Some people are hit by default values that cause more write consistency on replicas, transaction log writes, etc.

We use a lot of geo functionality and that received a very healthy performance boost.

Overall, I find that the server is a lot more stable and that is worth it by itself.

5

This will improve over time. We run nightly benchmarks of the latest master version. I also try to add numbers for the latest versions of past releases (i.e. 1.7.5 and 2.3.5 release).

However, performance can depend on a lot of factors and the best you can do is to set up an environment and run performance tests there with your own data. We develop and use Rally for our benchmarks and it can help you with some tasks related to benchmarking Elasticsearch.

Daniel