Kibana visualize network bandwidth from metribeat

[revolucio]

Hi all,

I'm trying to generate a network bandwidth visualization using Kibana 5.4.0

Now thanks to the derivative metric seems really easy so I did :

It works pretty well but from time to time I have a negative peak on the grapsh that really makes hard to understand what happen on the traffic:

So I'm almost crazy trying to find a solution.

The first idea is to only show positive data, but I have no idea how to do it.

Any idea on this will be more than welcome.

Thanks.

[shaunak (Shaunak Kashyap)]

system.network.in.bytes is a cumulative field, so you probably don't want to be summing it (since it already contains a sum). Instead try taking a max of it.

[revolucio]

Thanks a lot for the hint....

Max doesn't seem to solve the problem for me.

I will try to generate a new field that will be always positive for this.

Bests

[rcowart (Rob Cowart)]

You will probably have more success using Timelion in this case.

.es(index="metricbeat-*", metric="max:system.network.in.bytes", split="system.network.name:10", kibana=true).derivative().scale_interval(1s).if(operator="lt", if=0, then=0).trim(start=2,end=1).label(regex="^.* system.network.name:(.+) > .*$", label="$1").lines(width=2).yaxis(label="bytes / sec", min=0)

Lets break this down a bit...

.es(
  index="metricbeat-*",
  metric="max:system.network.in.bytes",
  split="system.network.name:10",
  kibana=true
)

Query the metricbeat-* index in Elasticsearch for the field system.network.in.bytes, returning values for up to 10 instances of interface names ( split="system.network.name:10" ). The chart should also apply Kibana timescales and filters ( kibana=true ).

.derivative()
Since the value is a counter we need the delta between successive buckets.

.scale_interval(1s)
We want the value expressed as the number of bytes per second.

.if(operator="lt", if=0, then=0)
If the value is less than zero, such as when the counter wraps, use zero instead. This will fix your negative spike issue.

.trim(start=2,end=1)
By removing the first two and last values we can remove a visual artifact that can occur due to partial time buckets at the beginning and end of the chart.

.label(regex="^.* system.network.name:(.+) > .*$", label="$1")
Extract the interface name from the auto-generated label and use that in the legend.

.lines(width=2)
I like a thinner line than the default, especially when displaying multiple interfaces on the same chart.

`.yaxis(label="bytes / sec", min=0)``
Finally, add a label to the y-axis to display the indicated units.

The resulting chart will be similar to this... (this config is a little different than above)

Rob

[revolucio]

Hi Robert,

I have no words for your reply, really useful and well explained.

I see that my original idea of draw this graph in a line visualisation is not as easy as it seems to me at the beginning.

Thank you, thank you very much....

[Sjaak01 (Sjaak)]

The new time series was touted as being able to make this easy but I'm running into the same issues as you. Derivatives produce incorrect results. How you can make a negative graph out of values that are always 0 or higher is beyond me.

In timelion I did it like this. Note that the mvavg requires a additional plugin. I needed this to smooth out my results for my netflow. Might not me necessary using metricbeat depending on how the data is collected.

$src_query='host:1.1.1.1, .es($src_query,metric='sum:netflow.in_bytes').mvavg(1m).scale_interval(1s).divide(1024).label('Up - KBps')

I made different queries for dst, total and also some separate subnets (host:1.1.1.1 AND src_add: 192.168.1.*) so I have 5 or 6 lines in one graph.

edit: scale.interval causes problems when you try to view data over a long period of time because it will create too many buckets. Use the mvavg mod instead.

[rcowart (Rob Cowart)]

@Sjaak01, I was about to reply to your related post with a link to what I posted here, but noticed that you found it already. I will also try out mvavg() myself. Thanks for the tip.