Indexing Rate Watch

Elastic Stack Elasticsearch
1

I would like to create a Watch for when the indexing rate for a specific index is 0 /s. How does the default monitoring dashboard Indexing Rate graph pull the data in? I am having trouble finding an API to call in the watch, as the index stats api does not seem to show timeseries data. Thanks!

2

hey,

you could reuse the monitoring indices that are being created by monitoring to do this. By using GET _cat/indices you should see the monitoring indices.

--Alex

3

Alex,

Does the monitoring index poll index stats?

Thanks

4

I was wrong here, it only contains stats about all indices.

So what about creating one watch, that gathers the stats via HTTP input, and queries a local index for the last entry and uses an index action to store in that local index, all you need is basically a timestamp and the docs count

GET .monitoring-es-*/_stats?filter_path=_all.primaries.docs

if you have deletes you might want to check for the sum of count and deleted

5

Ah, good idea. So I could collect stats for my index like that. How would you recommend calculating the rate/alerting? I found this blog post about anomaly detection. My desired alert is to tell me when my indexing rate is 0 docs / second for X units.

6

I think I got it.. I have one watch to poll the index _stats api that puts that data in another index. Then I have another watch that queries that index and does a serial_diff to see compare the document count between time windows. Not the prettiest, but it works

7

Hey Adam,

you could potentially do all of this in one watch, but for the sake of getting started and not suffering from the complexity, going with two watches and that approach sounds like a good idea!

--Alex

8

The Monitoring data does poll _stats for all indices.

GET /.monitoring-es-6-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "index_stats.index": "my-index123"
          }
        }
      ]
    }
  }
}

This will give you the last-fetched index stats for the index my-index123.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.