I want to do the max of the sum doc_count per second in line grpah visualization

Elastic Stack Kibana
1

Hello.
I'm trying to get the maximum value of the sum doc_count but using a max_bucket aggregation it either reaches the maximum number of buckets (error: "too_many_buckets_exception"), or it ajusts the timerange to be per hour and does not give me the value I want.

What I need is the equivalent of the "Normalize: per second" option but instead of getting the average, getting the maximum. So that if the visualization auto scales the horizontal axis to be my timefield per hour instead of per second, at least give me the maximum sum of documents that a second has in that hour.
For example if in an hour I have a second when the doc_count is 10 and in every other second of that hour the doc_count is smaller, I want the visualization to show 10 for that hour.

How can I do this?
I can use any type of visualization. If this is possible with any of the default visualization that would be prefered but if I need to do this with a custom plugin visualization can you tell me how to create a something like that please?

Thank you so much in advance.

2

Hi @Goncalo_Santos Welcome to the community

This isn’t really a visualization problem — it’s a bucketing problem. (as you already encountered ... there is no magic Viz for this)

What you want is:

  1. count logs at the 1-second level
  2. for each larger window, like 1 hour in a 24-hour view, return the maximum of those 1-second counts

The reason this gets difficult in a visualization is that once the chart auto-scales to larger intervals, it no longer has the original per-second buckets. And if you try to force per-second buckets across 24 hours, you can hit too_many_buckets_exception because that means evaluating up to 86,400 second buckets.

AND what makes it worse if you hack around to make this work... like increasing max_buckets you will most likely fail other ways ... or say 2 people try to run it at the same time... you will be sad :frowning:

That’s why the a different approach — using a transform — is the most likely best approach.

A transform lets you pre-aggregate the raw logs into a summary index, for example one document per second with a logs_per_second value. Then your visualization becomes simple: bucket those summary docs by hour and take the max(logs_per_second).

So the better pattern is:

  • Transform 1: create per-second counts from logs-*
  • Visualization/query: Then base your viz on the pre-aggregated data.

This is more scalable, avoids bucket-limit issues, and gives you the exact “peak second within the larger time window” value you’re after.

Transforms can have some side affects take resources etc but I would think this would be your first approach.

Here is a simple transform so you can get the ides

PUT \_transform/logs_per_second
{
  "source": {
    "index": ["logs-*"]
  },
  "dest": {
    "index": "logs_per_second"
  },
  "frequency": "5m",
  "sync": {
    "time": {
      "field": "@timestamp",
      "delay": "60s"
    }
  },
  "pivot": {
    "group_by": {
      "second": {
        "date_histogram": {
          "field": "@timestamp",
          "fixed_interval": "1s"
        }
      }
    },
    "aggregations": {
      "logs_per_second": {
        "value_count": {
          "field": "@timestamp"
        }
      }
    }
  }
}