Getting throttled by _msearch

Elastic Stack Elasticsearch
1

My scenario is as follows: I want to run a very large number of queries and I want to fully utilize my cluster (40 data nodes X 16 CPUs).

I am batching my queries (400 per batch) and sending them via _msearch, however it seems I'm getting throttled. The cluster CPUs hardly get utilized, and I simply cannot get past ~10 seconds for 400 queries, no matter how I play with the max_concurrent_searches and max_concurrent_shard_requests parameters. The took values per each query simply increase as I increase the concurrency, but the total time remains the same.

Any idea what I'm doing wrong here? I am using ES 6.5.3.

Thanks.

2

How large is your data set? Can it all fit in the operating system page cache? If not, it is quite possible that you are limited by disk I/O and not CPU.

3

The dataset doesn't fit in memory, however I don't think I'm I/O limited here. I've been I/O limited in ES before, and usually the nodes' load factor shoot through the roof. In this case it is staying pretty idle. In addition, the very same cluster is able to reach better throughput in production, when using the regular search endpoint, so it feels like this is related to _msearch.