Hot/Warm architecture and Index Stats

[prizov (Alexander Prizov)]

Hi guys!

Could you explain why index in Elasticsearch loose stats after relocation in case of changing routing allocation rules?

Steps to reproduce:
We have two zones hot and warm

1. Create index in hot zone
   curl -XPUT localhost:9200/test-index

2. Check that query total stats is equal 0
   curl -s -XGET localhost:9200/test-index/_stats | jq '.indices["test-index"].primaries.search.query_total'

3. Make some search request to test-index
   curl -XGET localhost:9200/test-index/_search

4. Check that query stats is changed
   curl -s -XGET localhost:9200/test-index/_stats | jq '.indices["test-index"].primaries.search.query_total'

5. Change allocation rule
   curl -X PUT "localhost:9200/test-index/_settings" -H 'Content-Type: application/json' -d '{ "index" : { "routing.allocation.require": {"zone": "cold"} } }'

6. Check stats -> It's flushed
   curl -s -XGET localhost:9200/test-index/_stats | jq '.indices["test-index"].primaries.search.query_total' = 0

If there is some workaround to save index stats after relocation?
Thanks!

[luiz.santos]

Hi @prizov,

According to our docs, it's an expected behavior:

Note, as shards move around the cluster, their stats will be cleared as they are created on other nodes. On the other hand, even though a shard "left" a node, that node will still retain the stats that shard contributed to.

If you have monitoring enabled, the following query will show you the value along the time (per day) and also the derivative (difference) from the last day:

GET .monitoring-es-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": {
        "term": {
          "type": "index_stats"
        }
      }
    }
  },
  "aggregations": {
    "index_name": {
      "terms": {
        "field": "index_stats.index",
        "size": 100
      },
      "aggs": {
        "per_day": {
          "date_histogram": {
            "field": "timestamp",
            "interval": "day"
          },
          "aggs": {
            "searches": {
              "max": {
                "field": "index_stats.primaries.search.query_total"
              }
            },
            "search_derivative": {
              "derivative": {
                "buckets_path": "searches"
              }
            }
          }
        }
      }
    }
  }
}

Result:

...
  "aggregations" : {
    "index_name" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : ".kibana_1",
          "doc_count" : 24952,
          "per_day" : {
            "buckets" : [
              {
                "key_as_string" : "2019-05-28T00:00:00.000Z",
                "key" : 1559001600000,
                "doc_count" : 2790,
                "searches" : {
                  "value" : 2088.0
                }
              },
              {
                "key_as_string" : "2019-05-29T00:00:00.000Z",
                "key" : 1559088000000,
                "doc_count" : 8640,
                "searches" : {
                  "value" : 2944.0
                },
                "search_derivative" : {
                  "value" : 856.0
                }
              },
...

I hope it helps!

Cheers,
Luiz Santos

[prizov (Alexander Prizov)]

Thanks!