1 of 4 shards failed

Hi all

I have a dashboard with some visualisations.

One visualisation seem be make a problem because I have the following message:

1 of 4 shards failed

She fail sometimes in fonction of my filters.

In my ES container log, I can see:

{"type": "server", "timestamp": "2019-07-02T14:03:27,045+0000", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "cloud-administration", "node.name": "cloud-administration-es-9c6nlqldbm", "cluster.uuid": "f9vtboJ9RQObv5_8hFTkMg", "node.id": "Ud4M-w0wR8qlxh24J5njFQ",  "message": "[apm-7.2.0-transaction-2019.07.02][0], node[rCQBD7NlSVyMmd7xWletPA], [P], s[STARTED], a[id=nXZXanuiSYWfYfQLDnKJGA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[apm-7.2.0-span-2019.07.02, apm-7.2.0-transaction-2019.07.02, apm-7.2.0-metric-2019.07.02, apm-7.2.0-error-2019.07.02], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='1562072294262', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=25, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={\"size\":0,\"timeout\":\"30000ms\",\"query\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"from\":\"2019-07-02T13:03:24.363Z\",\"to\":\"2019-07-02T14:03:24.364Z\",\"include_lower\":true,\"include_upper\":true,\"format\":\"strict_date_optional_time\",\"boost\":1.0}}}],\"filter\":[{\"match_all\":{\"boost\":1.0}},{\"match_all\":{\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"_source\":{\"includes\":[],\"excludes\":[]},\"stored_fields\":\"*\",\"docvalue_fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"},{\"field\":\"event.created\",\"format\":\"date_time\"},{\"field\":\"event.end\",\"format\":\"date_time\"},{\"field\":\"event.start\",\"format\":\"date_time\"},{\"field\":\"file.ctime\",\"format\":\"date_time\"},{\"field\":\"file.mtime\",\"format\":\"date_time\"},{\"field\":\"process.start\",\"format\":\"date_time\"}],\"script_fields\":{},\"track_total_hits\":2147483647,\"aggregations\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"time_zone\":\"Europe/Paris\",\"interval\":\"1m\",\"offset\":0,\"order\":{\"_key\":\"asc\"},\"keyed\":false,\"min_doc_count\":1},\"aggregations\":{\"3\":{\"terms\":{\"field\":\"service.environment\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}},\"4\":{\"terms\":{\"field\":\"service.name\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}},\"8\":{\"terms\":{\"field\":\"service.version\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}},\"5\":{\"terms\":{\"field\":\"transaction.name\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}},\"6\":{\"terms\":{\"field\":\"transaction.id\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}},\"7\":{\"terms\":{\"field\":\"transaction.duration.us\",\"size\":5,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"1\":\"desc\"},{\"_key\":\"asc\"}]},\"aggregations\":{\"1\":{\"cardinality\":{\"field\":\"@timestamp\"}}}}}}}}}}}}}}}}}}}]" , 
    "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [cloud-administration-es-22lzsf5mdp][10.2.1.113:9300][indices:data/read/search[phase/query]]",
    "Caused by: org.elasticsearch.search.aggregations.MultiBucketConsumerService$TooManyBucketsException: Trying to create too many buckets. Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max_buckets] cluster level setting.",
    "at org.elasticsearch.search.aggregations.MultiBucketConsumerService$MultiBucketConsumer.accept(MultiBucketConsumerService.java:110) ~[elasticsearch-7.2.0.jar:7.2.0]", [...]

Any idea ?

Thx for help

When the visualisation is in error:

{
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "1m",
        "time_zone": "Europe/Paris",
        "min_doc_count": 1
      },
      "aggs": {
        "3": {
          "terms": {
            "field": "service.environment",
            "order": {
              "1": "desc"
            },
            "size": 5
          },
          "aggs": {
            "1": {
              "cardinality": {
                "field": "@timestamp"
              }
            },
            "4": {
              "terms": {
                "field": "service.name",
                "order": {
                  "1": "desc"
                },
                "size": 5
              },
              "aggs": {
                "1": {
                  "cardinality": {
                    "field": "@timestamp"
                  }
                },
                "8": {
                  "terms": {
                    "field": "service.version",
                    "order": {
                      "1": "desc"
                    },
                    "size": 5
                  },
                  "aggs": {
                    "1": {
                      "cardinality": {
                        "field": "@timestamp"
                      }
                    },
                    "5": {
                      "terms": {
                        "field": "transaction.name",
                        "order": {
                          "1": "desc"
                        },
                        "size": 5
                      },
                      "aggs": {
                        "1": {
                          "cardinality": {
                            "field": "@timestamp"
                          }
                        },
                        "6": {
                          "terms": {
                            "field": "transaction.id",
                            "order": {
                              "1": "desc"
                            },
                            "size": 5
                          },
                          "aggs": {
                            "1": {
                              "cardinality": {
                                "field": "@timestamp"
                              }
                            },
                            "7": {
                              "terms": {
                                "field": "transaction.duration.us",
                                "order": {
                                  "1": "desc"
                                },
                                "size": 5
                              },
                              "aggs": {
                                "1": {
                                  "cardinality": {
                                    "field": "@timestamp"
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "size": 0,
  "_source": {
    "excludes": []
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "event.created",
      "format": "date_time"
    },
    {
      "field": "event.end",
      "format": "date_time"
    },
    {
      "field": "event.start",
      "format": "date_time"
    },
    {
      "field": "file.ctime",
      "format": "date_time"
    },
    {
      "field": "file.mtime",
      "format": "date_time"
    },
    {
      "field": "process.start",
      "format": "date_time"
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2019-07-02T13:07:20.256Z",
              "lte": "2019-07-02T14:07:20.256Z"
            }
          }
        }
      ],
      "filter": [
        {
          "match_all": {}
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

When the visualisation works:

{
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "30m",
        "time_zone": "Europe/Paris",
        "min_doc_count": 1
      },
      "aggs": {
        "3": {
          "terms": {
            "field": "service.environment",
            "order": {
              "1": "desc"
            },
            "size": 5
          },
          "aggs": {
            "1": {
              "cardinality": {
                "field": "@timestamp"
              }
            },
            "4": {
              "terms": {
                "field": "service.name",
                "order": {
                  "1": "desc"
                },
                "size": 5
              },
              "aggs": {
                "1": {
                  "cardinality": {
                    "field": "@timestamp"
                  }
                },
                "8": {
                  "terms": {
                    "field": "service.version",
                    "order": {
                      "1": "desc"
                    },
                    "size": 5
                  },
                  "aggs": {
                    "1": {
                      "cardinality": {
                        "field": "@timestamp"
                      }
                    },
                    "5": {
                      "terms": {
                        "field": "transaction.name",
                        "order": {
                          "1": "desc"
                        },
                        "size": 5
                      },
                      "aggs": {
                        "1": {
                          "cardinality": {
                            "field": "@timestamp"
                          }
                        },
                        "6": {
                          "terms": {
                            "field": "transaction.id",
                            "order": {
                              "1": "desc"
                            },
                            "size": 5
                          },
                          "aggs": {
                            "1": {
                              "cardinality": {
                                "field": "@timestamp"
                              }
                            },
                            "7": {
                              "terms": {
                                "field": "transaction.duration.us",
                                "order": {
                                  "1": "desc"
                                },
                                "size": 5
                              },
                              "aggs": {
                                "1": {
                                  "cardinality": {
                                    "field": "@timestamp"
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "size": 0,
  "_source": {
    "excludes": []
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "event.created",
      "format": "date_time"
    },
    {
      "field": "event.end",
      "format": "date_time"
    },
    {
      "field": "event.start",
      "format": "date_time"
    },
    {
      "field": "file.ctime",
      "format": "date_time"
    },
    {
      "field": "file.mtime",
      "format": "date_time"
    },
    {
      "field": "process.start",
      "format": "date_time"
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2019-07-01T14:11:17.075Z",
              "lte": "2019-07-02T14:11:17.075Z"
            }
          }
        }
      ],
      "filter": [
        {
          "match_all": {}
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.