Aggregation problem with timeseries - terms / sum

[Thomas74 (Thomas R.)]

Hi,

I'm collecting metrics of my cluster via prometheus and metricbeat. Documents received are timeseries. That's mean I have multiple document for the same timestamp.

Each 10s I have a list of all index with their consumption and tagged with their group.

Here are simple example of documents :

{
         "prometheus" : {
            "query" : {
              "elasticsearch_indices_store_size_bytes_total" : 1.07419744767E11
            },
            "labels" : {
              "job" : "elasticsearch_exporter",
              "__name__" : "elasticsearch_indices_store_size_bytes_total",
              "cluster" : "elasticsearch",
              "index" : "filebeat_proxy-000131",
              "instance" : "xxx"
            }
          },
          "service" : {
            "type" : "prometheus",
            "address" : "xxx"
          },
          "index.agent" : "filebeat",
          "@version" : "1",
          "index.space" : "proxy",
          "ecs" : {
            "version" : "1.6.0"
          },
          "host" : {
            "name" : "mmm"
          },
          "agent" : {
            "id" : "e61d116f-519c-49f9-b488-45396db2a87e",
            "ephemeral_id" : "8d46f006-73bb-48ae-a61f-9422e292a191",
            "version" : "7.10.0",
            "name" : "mmm",
            "hostname" : "mmm",
            "type" : "metricbeat"
          },
          "index.number" : "000131",
          "@timestamp" : "2021-01-08T14:18:00.000Z",
},
{
         "prometheus" : {
            "query" : {
              "elasticsearch_indices_store_size_bytes_total" : 1.07419744767E11
            },
            "labels" : {
              "job" : "elasticsearch_exporter",
              "__name__" : "elasticsearch_indices_store_size_bytes_total",
              "cluster" : "elasticsearch",
              "index" : "filebeat_proxy-000132",
              "instance" : "xxx"
            }
          },
          "service" : {
            "type" : "prometheus",
            "address" : "xxx"
          },
          "index.agent" : "filebeat",
          "@version" : "1",
          "index.space" : "proxy",
          "ecs" : {
            "version" : "1.6.0"
          },
          "host" : {
            "name" : "mmm"
          },
          "agent" : {
            "id" : "e61d116f-519c-49f9-b488-45396db2a87e",
            "ephemeral_id" : "8d46f006-73bb-48ae-a61f-9422e292a191",
            "version" : "7.10.0",
            "name" : "mmm",
            "hostname" : "mmm",
            "type" : "metricbeat"
          },
          "index.number" : "000132",
          "@timestamp" : "2021-01-08T14:18:00.000Z",
}

I want to visualize storage consumption per index.space with a bar graph. I tried aggregations to have this metric but with no success. As they are timeseries, I don't see how to do this.

The steps I think I have to do :

• Get top 1 last timestamp
• Aggregate term index.space
• Aggratate sum of prometheus.query.elasticsearch_indices_store_size_bytes_total

The problem is I can't top 1 a timestamp.

Do you have any idea to do the query and how to deal with timeseries ?

I tried with this query but it is not what I want :

GET metricbeat_prometheus/_search?size=0
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "prometheus.labels.job": "elasticsearch_exporter"
          }
        },
        {
          "exists": {
            "field": "prometheus.query.elasticsearch_indices_store_size_bytes_total"
          }
        },
        {
          "exists": {
            "field": "index.space"
          }
        }
      ],
      "must_not": [
        {
          "match": {
            "tags": "_grokparsefailure" 
          }
        }
      ]
    }
  },
  "aggs": {
    "unique_indexes": {
      "terms": {
        "field": "prometheus.labels.index",
        "size": 100
      },
      "aggs": {
        "top": {
          "top_hits": {
            "_source": {"includes":[ "prometheus.query.elasticsearch_indices_store_size_bytes_total", "index.space" ]}, 
            "size": 1
          }
        }
      }
    }
  }
}

Best regards,

Thomas