2-phase aggregation on Y-axis value?


I have a requirement below and want some advise how to do this.

  • Target log is kind of summarized access log which has fields like timestamp, click_count, url, some_parameter. Interval of timestamp is usually 10 seconds.
    For example:
    00:00:10, 10 , /a, p1
    00:00:11, 5 , /a, p2
    00:00:20, 3 , /a, p1
    00:00:21, 6 , /a, p2
    00:01:10, 4, /a, p1
    00:01:10, 8, /a, p2

  • I want to visualize (on Kibana) sum of total click count per minute, but if we have multiple record for the same "url" and "some_parameter" in the same minute, that value should be average.

In SQL, the aggregation can be like this:

SELECT  -- The values to be plotted are sum(click_count) by URL
  timestamp, sum(click_count), url
  SELECT   -- calculate average 1 minute click_count by 'url' and 'some_parameter'
     round(timestamp,'min') as timestamp ,  avg(click_count), url, some_parameter
     round(timestamp,'min'), url, some_parameter
    url, timestamp

As far as I know, I can specify only one aggregation type for Y-axis, but what I want to do is kind of 2-phase aggregation, first do 'average' aggregation in one minute by url and some_parameter, and then do 'sum' aggregation by url.

Are there any way to do this on Kibana?

Hironori Ogibayashi

A mockup of the type of visualization you're trying to accomplish based on the dataset above would help.

Here is a visualization of my data.

I plotted 1 min. average click_count, split lines by uri and split chart by param. The raw log data is generated every 10 seconds.
What I really want is plotting the total value of click_count by url (click_count for param p1 + p2) , looks like this.

The problem is that: if I specify "Average" to the aggregation of Y-axis, the value will be the average of all the value in that 1 minute regardless of "param". I want values calculated by first taking average for values in 1 minute by each url and param , then sum up the values by url.

Does this make sense?