Metric to count number of queries per day/month

Elastic Stack Elasticsearch
1

Hi,

I'm trying to collect the number of queries that users send to Elasticsearch to understand how many queries per day/month are submitted to our clusters.
I've tried using the following metrics mentioned in Nodes stats API | Elasticsearch Guide [8.6] | Elastic
indices.search.fetch_total
indices.search.query_total

However, I'm not sure what the two above metrics mean exactly.
I'm using Elastic integration of Datadog to collect those metrics (submitted as elasticsearch.search.fetch.total.count and elasticsearch.search.query.total.count). My idea was to aggregate the count using cumulative sum during a given period of time (1 day or 1 month) and take the last value to represent how many queries were submitted in that period.

I ran a few small experiments with a few queries on a test cluster and here is what I understood:

elasticsearch.search.query.total.count seems to count the number of docs returned by queries, which is not what we want, but the metric name is confusing.

elasticsearch.search.fetch.total.count seems to count the number of queries, which is probably what we want, but there is a problem:

An Elasticsearch cluster with not a lot of active queries shows really large numbers for elasticsearch.search.fetch.total.count (up to 700,000,000 per month) which is surprising and probably wrong.

Am I missing something?

Thanks

2

Hello @Milad_Heydariaan , welcome to the community !
I believe the stats you are referring to are correct and _node/stats/indices/query_total should, by definition, provide total query operations. Have you tried the command in DevConsole and compare the result with elasticsearch.search.query.total.count ?
I'd suggest you to setup a scripted query which directly works with _node/stats rather than relying on Datadog or any other third party API since ES provides what you need here.

3

Hi @Ayush_Mathur
I tried testing the method you suggested with a test cluster that runs on 2 data nodes. I have an index named test, with 3 shards, 1 replica, and 2 documents.
This is what I see:

  • Checked _nodes/stats/indices first and I see:
    .nodes.*.indices.search.query_total shows 339 and 6851 (Total: 7190)
    .nodes.*.indices.search.fetch_total shows 7 and 761 (Total: 768)
  • Ran GET test/_search on Kibana Console and I get:
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test",
        "_type" : "_doc",
        "_id" : "UyNs24IBF2DHLrbLO01N",
        "_score" : 1.0,
        "_source" : {
          "number" : 10
        }
      },
      {
        "_index" : "test",
        "_type" : "_doc",
        "_id" : "Jwts24IBfmjNTyhJVVal",
        "_score" : 1.0,
        "_source" : {
          "number" : 11
        }
      }
    ]
  }
}
  • Checked _nodes/stats/indices again and I see:
    .nodes.*.indices.search.query_total shows 339 and 6854 (Total: 7193)
    .nodes.*.indices.search.fetch_total shows 7 and 763 (Total: 770)

This is surprising to me since query_total was increased by 3 and fetch_total was increased by 2. I was expecting to see the query_total increasing just by 1 since I only ran 1 query.
My goal is to see how many queries are flowing to our cluster and report aggregated statistics daily/monthly.

If query_total or fetch_total are not suitable for what I need, is there another metric that I can use?

Another question: are those metrics increasing for the lifetime of the cluster or they reset to 0 at some point?

4

Ok, so the query_total increased by 3 means all 3 shards were included in your search query for data consolidation. In other words, query_total would provide the summation of number of shards queried for your request since the cluster was last started.
The fetch_total on the other hand provides the total hits count to your search query that were aggregated for response.
I would suggest to look into auditing of ES and Kibana and capture the audit logs in ES (using filebeat): Collecting Elasticsearch log data with Filebeat | Elasticsearch Guide [8.6] | Elastic
It provides some fields under the hood like origin.type, user.run_as_name, action, etc. that can be further used to create scripts and visualizations. For fields reference, read here: Elasticsearch fields | Filebeat Reference [8.6] | Elastic
In addition, there are few properties exposed for this as mentioned in PR: Enable logging in ES client by albertteoh · Pull Request #2862 · jaegertracing/jaeger · GitHub
And some solution provided in StackOverflow: debugging - How to log all executed elasticsearch queries - Stack Overflow

Furthermore, if you would like to log ES queries executed from Kibana, you can refer: Examples | Kibana Guide [8.6] | Elastic

5

So, it seems there is no way to directly get this info from ES. I'll try the log-based solution you suggested. Thanks!

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.