Here are some slow logs when ES timed out while searching for 12 hours of APM logs
{"type": "index_search_slowlog", "timestamp": "2020-03-16T19:09:47,438Z", "level": "WARN", "component": "i.s.s.query", "cluster.name": "elasticsearch", "node.name": "elasticsearch-es-data-4", "message": "[apm-7.6.0-2020.03.16][3]", "took": "45.5s", "took_millis": "45508", "total_hits": "10000+ hits", "stats": "[]", "search_type": "QUERY_THEN_FETCH", "total_shards": "12", "source": "{\"size\":0,\"query\":{\"bool\":{\"filter\":[{\"terms\":{\"processor.event\":[\"transaction\",\"error\",\"metric\"],\"boost\":1.0}},{\"range\":{\"@timestamp\":{\"from\":1584342540477,\"to\":1584385740477,\"include_lower\":true,\"include_upper\":true,\"format\":\"epoch_millis\",\"boost\":1.0}}},{\"range\":{\"observer.version_major\":{\"from\":7,\"to\":null,\"include_lower\":true,\"include_upper\":true,\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"aggregations\":{\"host\":{\"meta\":{},\"filter\":{\"match_all\":{\"boost\":1.0}},\"aggregations\":{\"by_terms\":{\"terms\":{\"field\":\"host.hostname\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"bucket_count\":{\"cardinality\":{\"field\":\"service.name\"}}}}}},\"agentName\":{\"meta\":{},\"filter\":{\"match_all\":{\"boost\":1.0}},\"aggregations\":{\"by_terms\":{\"terms\":{\"field\":\"agent.name\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"bucket_count\":{\"cardinality\":{\"field\":\"service.name\"}}}}}}}}", "cluster.uuid": "jktKncXUQ_yiDkjUepqUCg", "node.id": "yCTfEa-nSJi5QhH-W8Praw" }
{"type": "index_search_slowlog", "timestamp": "2020-03-16T19:09:47,424Z", "level": "WARN", "component": "i.s.s.query", "cluster.name": "elasticsearch", "node.name": "elasticsearch-es-data-4", "message": "[apm-7.6.0-2020.03.16][3]", "took": "45.4s", "took_millis": "45426", "total_hits": "10000+ hits", "stats": "[]", "search_type": "QUERY_THEN_FETCH", "total_shards": "12", "source": "{\"size\":0,\"query\":{\"bool\":{\"filter\":[{\"terms\":{\"processor.event\":[\"transaction\",\"error\",\"metric\"],\"boost\":1.0}},{\"range\":{\"@timestamp\":{\"from\":1584342540477,\"to\":1584385740477,\"include_lower\":true,\"include_upper\":true,\"format\":\"epoch_millis\",\"boost\":1.0}}},{\"range\":{\"observer.version_major\":{\"from\":7,\"to\":null,\"include_lower\":true,\"include_upper\":true,\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"aggregations\":{\"services\":{\"terms\":{\"field\":\"service.name\",\"size\":500,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"avg\":{\"avg\":{\"field\":\"transaction.duration.us\"}},\"agents\":{\"terms\":{\"field\":\"agent.name\",\"size\":1,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_key\":\"asc\"}]}},\"events\":{\"terms\":{\"field\":\"processor.event\",\"size\":2,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_key\":\"asc\"}]}},\"environments\":{\"terms\":{\"field\":\"service.environment\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_key\":\"asc\"}]}}}}}}", "cluster.uuid": "jktKncXUQ_yiDkjUepqUCg", "node.id": "yCTfEa-nSJi5QhH-W8Praw" }
AS you can see in the screenshots below CPU is not spiked at all, search queue is completely empty and there are no rejects in the queue.