Elasticsearch Indexing Rate - API

Elastic Stack Elasticsearch
1

Hi,

I'm trying to calculate the Indexing rate for our cluster using the API. I'm basically trying to replicate the data from the "Monitoring" graphs for our external alerting system. As you can see these are showing an indexing rate of ~3k per second

2019-11-25_13-17
2019-11-25_13-17.png900×284 32.3 KB

A similar question has been asked before so following the advise here I have ran the following:

GET _stats?pretty:

  "indexing" : {
    "index_total" : 1791502388,
    "index_time_in_millis" : 1339241001,
    "index_current" : 0,
    "index_failed" : 1,
    "delete_total" : 9883899,
    "delete_time_in_millis" : 681911,
    "delete_current" : 0,
    "noop_update_total" : 0,
    "is_throttled" : false,
    "throttle_time_in_millis" : 0
  },

Wait 60 seconds then re-run:

  "indexing" : {
    "index_total" : 1791564251,
    "index_time_in_millis" : 1339278670,
    "index_current" : 0,
    "index_failed" : 1,
    "delete_total" : 9883899,
    "delete_time_in_millis" : 681911,
    "delete_current" : 0,
    "noop_update_total" : 0,
    "is_throttled" : false,
    "throttle_time_in_millis" : 0
  }

Using the calculations given in the above linked answer I get an Indexing rate of 1031/s (1791564251-1791502388)/60

How can I get the same numbers from the graphs above using the API? Is this even possible?

2

Hi,

your number seems to match the primary shards metrics of ~ 1100 docs/s pretty well. Quoting from the index stats API docs:

primaries are the values for only the primary shards. total are the accumulated values for both primary and replica shards.

So I assume that the numbers you got are from the primaries but not from total.

With regards to the difference you also need to account for the different sampling rate (see xpack.monitoring.collection.interval in our docs) which is 10 seconds by default. These two adjustments should bring you pretty close*) to the numbers you see in the monitoring application.

Daniel

*) you can't get the exact same values as you'd need to sample at the exact same point in time as x-pack monitoring but that shouldn't be a problem in practice.

1 Like
3

Ah! You are indeed correct @danielmitterdorfer I was looking at the primaries rather than the total. Thanks, I feel a bit silly now.

4

Don't. Such things can happen so easily. :slight_smile: I'm glad we could sort it out.

Daniel

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.