Kibana/Netflow bandwidth percentile calculation

Elastic Stack Kibana
1

Hello, and thanks in advance for any help.

I'm working on a bandwidth usage charting tool where I need to get a monthly percentile calculation, and am most of the way there - Here's what I have so far:

An Elastic Stack receiving Netflow data from a pfSense router. I have the first calculation that I need done, here's a screenshot of the result. For purposes of explaining my problem, I've zoomed way in to 5x points in my graph:

05
Screen Shot 2019-03-11 at 20.40.05.png2986×1706 248 KB

I have two plots on that graph - The blue one is exactly what I want - It's the sum of all data usage in 5 minute intervals.

For the percentile calculation, the red line is as close as I've gotten, and it helps highlight the issue - It's taking the 95th percentile of the individual netflow entries - What I need is the 95th percentile of the 5-minute "buckets", or the points that make up the blue line.

95th percentile gets a bit weird for a plot with only 5 points, so lets say I was looking for the 75th percentile - In this example, I'd be looking for "384387", which is the 2nd highest value out of five.

On continuing to work on this, I was able to create a new visualization (Data Table type) that also makes me feel like I'm really close. It's basically the same thing as my TimeLion graph, but as a table. This also highlights the issue in a similar way: It's very easy to get a percentile of the individual values that make up each Bucket, but getting a percentile of the Buckets themselves proves challenging:

16
Screen Shot 2019-03-12 at 10.35.16.png2998×1150 277 KB

So far, I haven't found any way to accomplish this with Kibana and TimeLion. Any help would be greatly appreciated!

EDIT: one more clarification I thought I should add is that where I'm going with this is having 1 fixed percentile per month (and then down the road it will be one percentile per month per source IP, but I think I'm already good on knowing how to break it out by that). That percentile will be calculated from all of the 5-minute buckets for that month, and ideally I can just put that single value on the chart as a horizontal line.

here's the code from the first screenshot:

.elasticsearch(
    index="netflow-*", 
    metric="sum:netflow.bytes",
    split="netflow.src_addr:1",
    kibana=true)
.label(
    regex="^.* netflow.src_addr:(.+) > .*$",
    label="$1")
.yaxis(
    label="bytes / sec",
    min=0)
,
.elasticsearch(
    index="netflow-*", 
    metric="percentiles:netflow.bytes:95",
    split="netflow.src_addr:1",
    kibana=true)
.label(
    regex="^.* netflow.src_addr:(.+) > .*$",
    label="$1")
2

On reviewing my post, I do see that a couple things changed between taking the first screenshot and pasting the code at the end. Either code should produce pretty much the same result, I just changed the labels to help clarify what the chart was showing.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.