Kibana search for last 24h wait for warm nodes

[mkostrikin (Maxim N Kostrikin)]

Hello,
We have a ELK stack with hot/warm ES nodes, 3 node for hot and 3 node for warm.
We keep 7 days in hot, older moved to warm.
We expect searches for last 24 hour will load only hot ES nodes, but kibana keep waiting for reply ( we guess ) from warm ES nodes. Load of ES nodes are monitored and one of the warm node is 100% CPU busy, during test searches.
ES 2.2.2 version, Kibana 4.4.0
Searching for similar issues gives nothing.

Please advise.
Thanks,

[Christian_Dahlqvist]

How have you specified your index patterns in Kibana?

[mkostrikin (Maxim N Kostrikin)]

Settings for index patterns are:
Index contains time-based events - CHECKED
Use event times to create index names - UNCHECKED
logstash-*
Do not expand index pattern when searching - UNCHECKED
Time-field name "@timestamp"

[Christian_Dahlqvist]

I am not sure if Kibana 4.4 was able to limit the amount of indices queried based on the field stats API. so by using a wildcard pattern it is possible that you are hitting all shards. For earlier versions of Kibana, before the field stats API was available, it was generally beneficial to configure the date pattern in the Kibana index pattern.

[mkostrikin (Maxim N Kostrikin)]

I have tried logstash-* , with and without Do not expand index pattern when searching
[logstash-]YYYY.MM.DD
All are loading warm nodes by search on last 24h.

I am going to try same searches on 2.4 or 5.1 ES cluster.

[Christian_Dahlqvist]

You will need to enable Use event times to create index names and leave Do not expand index pattern when searching disabled. You can use Chrome developer tools (or similar functionality in other browsers) to inspect the request and verify that it is rewritten to only target a subset of the available indices.

[mkostrikin (Maxim N Kostrikin)]

{"index":["logstash-*"],"ignore_unavailable":true}
{"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":"*"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"gte":1484719730791,"lte":1484806130791,"format":"epoch_millis"}}}],"must_not":[]}}}},"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"*":{}},"require_field_match":false,"fragment_size":2147483647},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"30m","time_zone":"UTC","min_doc_count":0,"extended_bounds":{"min":1484719730786,"max":1484806130786}}}},"fields":["*","_source"],"script_fields":{},"fielddata_fields":["@timestamp","timestamp8601"]}

This is payload for search last 24h.
It does not select any indeces. I guess, kibana selects indeces for search.

As I mentioned before, [logstash-]YYYY.MM.DD with enabled Use event times to create index names and disabled Do not expand index pattern when searching. Last one hidden with Use event... - gives the same problems. warm nodes are got loaded.