Hi all
I have a dashboard with some visualisations.
One visualisation seem be make a problem because I have the following message:
1 of 4 shards failed
She fail sometimes in fonction of my filters.
In my ES container log, I can see:
{"type": "server", "timestamp": "2019-07-02T14:03:27,045+0000", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "cloud-administration", "node.name": "cloud-administration-es-9c6nlqldbm", "cluster.uuid": "f9vtboJ9RQObv5_8hFTkMg", "node.id": "Ud4M-w0wR8qlxh24J5njFQ", "message": "[apm-7.2.0-transaction-2019.07.02][0], node[rCQBD7NlSVyMmd7xWletPA], [P], s[STARTED], a[id=nXZXanuiSYWfYfQLDnKJGA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[apm-7.2.0-span-2019.07.02, apm-7.2.0-transaction-2019.07.02, apm-7.2.0-metric-2019.07.02, apm-7.2.0-error-2019.07.02], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='1562072294262', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=25, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={\"size\":0,\"timeout\":\"30000ms\",\"query\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"from\":\"2019-07-02T13:03:24.363Z\",\"to\":\"2019-07-02T14:03:24.364Z\",\"include_lower\":true,\"include_upper\":true,\"format\":\"strict_date_optional_time\",\"boost\":1.0}}}],\"filter\":[{\"match_all\":{\"boost\":1.0}},{\"match_all\":{\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"_source\":{\"includes\":[],\"excludes\":[]},\"stored_fields\":\"*\",\"docvalue_fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"},{\"field\":\"event.created\",\"format\":\"date_time\"},{\"field\":\"event.end\",\"format\":\"date_time\"},{\"field\":\"event.start\",\"format\":\"date_time\"},{\"field\":\"file.ctime\",\"format\":\"date_time\"},{\"field\":\"file.mtime\",\"format\":\"date_time\"},{\"field\":\"process.start\",\"format\":\"date_time\"}],\"script_fields\":{},\"track_total_hits\":2147483647,\"aggregations\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"time_zone\":\"Europe/Paris\",\"interval\":\"1m\",\"offset\":0,\"order\":{\"_key\":\"asc\"},\"keyed\":false,\"min_doc_count\":1},\"aggregations\":{\"3\":{\"terms\":{\"field\":\"service.environment\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}},\"4\":{\"terms\":{\"field\":\"service.name\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}},\"8\":{\"terms\":{\"field\":\"service.version\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}},\"5\":{\"terms\":{\"field\":\"transaction.name\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}},\"6\":{\"terms\":{\"field\":\"transaction.id\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}},\"7\":{\"terms\":{\"field\":\"transaction.duration.us\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}}}}}}}}}}}}}}}}}}}]" ,
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [cloud-administration-es-22lzsf5mdp][10.2.1.113:9300][indices:data/read/search[phase/query]]",
"Caused by: org.elasticsearch.search.aggregations.MultiBucketConsumerService$TooManyBucketsException: Trying to create too many buckets. Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max_buckets] cluster level setting.",
"at org.elasticsearch.search.aggregations.MultiBucketConsumerService$MultiBucketConsumer.accept(MultiBucketConsumerService.java:110) ~[elasticsearch-7.2.0.jar:7.2.0]", [...]
Any idea ?
Thx for help