Visualize difference with offset

Elastic Stack Kibana
1

Dear,

We am struggling with an important visualisation for our team, not able to find the best way so I hope anybody can help me. It is related to thread pools elastic statistic, to give you an idea we can consider a simple data format, where reach record in an index has just node name and total number of rejected requests (it is a cumulative number so it only increments).

We would like to be able to see for which 5 nodes the number of rejected requests changed the most over last week or day.

One way we tried is using serial diff pipeline aggregation, but it works just over time and I am able to see are many changes of the count for the time pieces. Ideally we would need to do a sum over the aggregation, but it is not allowed.

19
Screen Shot 2018-05-25 at 17.03.19.png2420×478 29 KB

We were also thinking about using a scripted field to create the serial difference on my own, but to do so, i would need to access other records using something like top hit when executing such script and did not find a way to do that.

If anybody has an idea how to do that, it would be awesome

Thx,
Ondra

2

For this example, I am using documents which look like this, where node is the node name, and failed is the cumulative rejected requests:

{
  "node": "node2",
  "failed": 6,
  "@timestamp": 1527267655
}

Take a look at this query:

{
  "size": 0,
  "aggs": {
    "nodes": {
      "terms": {
        "field": "node.keyword"
      },
      "aggs": {
        "min_failures": {
          "min": {
            "field": "failed"
          }
        },
        "max_failures": {
          "max": {
            "field": "failed"
          }
        },
        "failure_diff": {
          "bucket_script": {
            "buckets_path": {
              "min_failures": "min_failures",
              "max_failures": "max_failures"
            },
            "script": "params.max_failures - params.min_failures"
          }
        },
        "failure_diff_sort": {
          "bucket_sort": {
            "sort": [
              {
                "failure_diff": {
                  "order": "desc"
                }
              }
            ],
            "size": 5
          }
        }
      }
    }
  }
}

Which produces this result:

{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 4,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "nodes": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "node1",
          "doc_count": 2,
          "max_failures": {
            "value": 10
          },
          "min_failures": {
            "value": 1
          },
          "failure_diff": {
            "value": 9
          }
        },
        {
          "key": "node2",
          "doc_count": 2,
          "max_failures": {
            "value": 6
          },
          "min_failures": {
            "value": 5
          },
          "failure_diff": {
            "value": 1
          }
        }
      ]
    }
  }
}

This query uses aggregations to:

The key here is being able to use Bucket Script Aggregations, which is not yet supported in Kibana visualizations. There is an open issue here: https://github.com/elastic/kibana/issues/4707. But, I hope this query helps.

3

bucket script is available in Visual Builder visualization.

4

Many thanks for your quick response. It is nice way to use the bucket scripts, but I am not sure where to place it in the Visual Builder, is there a tutorial how to use it?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.