Kibana Monitoring Nodes - There are no records that match your query

[rugenl (Len Rugen)]

I set search.max_buckets=10000 because our cluster was breaking at times when we got these messages in the log.

After that change, I went to the monitoring screen listing the nodes, it worked for "last 15 minutes" gets:
"no records that match" for "last 1 hour" or longer.

Version 6.3.1

[chrisronline (Chris Roberson)]

Hi @rugenl,

Can you open the network tab in the developer tools and find the cluster request and paste the entire request/response here?

[rugenl (Len Rugen)]

Request URL: https://...missouri.edu/api/monitoring/v1/clusters/DWNq4pOHTOuOfdYShnhaig/elasticsearch/nodes
Request Method: POST
Status Code: 200 OK
Remote Address: xxx.xxx.xxx.xxx:443
Referrer Policy: no-referrer-when-downgrade
cache-control: no-cache
Connection: Keep-Alive
content-encoding: gzip
content-type: application/json; charset=utf-8
Date: Mon, 06 Aug 2018 23:21:44 GMT
kbn-name: kibana
kbn-version: 6.3.2
kbn-xpack-sig: b222b911b1ef01c0b10618e8215698f5
Keep-Alive: timeout=5, max=99
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
Transfer-Encoding: chunked
vary: accept-encoding
Accept: application/json, text/plain, /
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Authorization: Basic bWV0cmljc3RlYW06aGlkZGVuRUxL
Connection: keep-alive
Content-Length: 81
Content-Type: application/json;charset=UTF-8
Cookie: _ga=GA1.2.1804478381.1511274427; __utma=262015377.1804478381.1511274427.1525787649.1525787649.1; __utmz=262015377.1525787649.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)
Host: ...missouri.edu
kbn-version: 6.3.2
Origin: https://...missouri.edu
Referer: https://....missouri.edu/app/monitoring
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 Safari/537.36
{"timeRange":{"min":"2018-08-06T22:21:43.608Z","max":"2018-08-06T23:21:43.609Z"}}

{"clusterStatus":{"status":"green","indicesCount":110,"documentCount":2663868064,"dataSize":2671062360767,"nodesCount":14,"upTime":292134182,"version":["6.3.2"],"memUsed":77090275992,"memMax":274076008448,"unassignedShards":0,"totalShards":564},"nodes":[]

[rugenl (Len Rugen)]

It gets the max_buckets error

[2018-08-06T19:14:40,613][WARN ][r.suppressed ] path: /.monitoring-es-2-%2C.monitoring-es-6-/_search, params: {size=10000, ignore_unavailable=true, index=.monitoring-es-2-,.monitoring-es-6-, filter_path=hits.total,hits.hits._source.source_node,aggregations.nodes.buckets.key,aggregations.nodes.buckets.node_cgroup_quota.buckets,aggregations.nodes.buckets.node_cgroup_throttled.buckets,aggregations.nodes.buckets.node_cpu_utilization.buckets,aggregations.nodes.buckets.node_load_average.buckets,aggregations.nodes.buckets.node_jvm_mem_percent.buckets,aggregations.nodes.buckets.node_free_space.buckets}
org.elasticsearch.action.search.SearchPhaseExecutionException:
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:91) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:710) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.2.jar:6.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: org.elasticsearch.search.aggregations.MultiBucketConsumerService$TooManyBucketsException: Trying to create too many buckets. Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max_buckets] cluster level setting.