Kibana monitoring gaps when data node is replaced

Elastic Stack Kibana
5

Hmm. Okay, let's figure this out.

Let's run the query to fetch the Search Rate graph for one of these black-out periods and see what the data is telling us:

Fill in the <cluster_uuid> with the right cluster and then adjust the time period to the affected time period.

POST .monitoring-es-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "cluster_uuid": "<cluster_uuid>"
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": "2019-12-16T00:51:07.080Z",
              "lte": "2019-12-16T18:51:07.080Z"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "check": {
      "date_histogram": {
        "field": "timestamp",
        "fixed_interval": "30s"
      },
      "aggs": {
        "metric": {
          "max": {
            "field": "indices_stats._all.total.search.query_total"
          }
        },
        "metric_deriv": {
          "derivative": {
            "buckets_path": "metric",
            "gap_policy": "skip",
            "unit": "1s"
          }
        }
      }
    }
  }
}

Let's see if this helps us. Thanks!