Kibana visualize network bandwidth from metribeat


(revolucio) #1

Hi all,

I'm trying to generate a network bandwidth visualization using Kibana 5.4.0

Now thanks to the derivative metric seems really easy so I did :

It works pretty well but from time to time I have a negative peak on the grapsh that really makes hard to understand what happen on the traffic:

So I'm almost crazy trying to find a solution.

The first idea is to only show positive data, but I have no idea how to do it.

Any idea on this will be more than welcome.

Thanks.


(Shaunak Kashyap) #2

system.network.in.bytes is a cumulative field, so you probably don't want to be summing it (since it already contains a sum). Instead try taking a max of it.


(revolucio) #3

Thanks a lot for the hint....

Max doesn't seem to solve the problem for me.

I will try to generate a new field that will be always positive for this.

Bests


(Robert Cowart) #4

You will probably have more success using Timelion in this case.

.es(index="metricbeat-*", metric="max:system.network.in.bytes", split="system.network.name:10", kibana=true).derivative().scale_interval(1s).if(operator="lt", if=0, then=0).trim(start=2,end=1).label(regex="^.* system.network.name:(.+) > .*$", label="$1").lines(width=2).yaxis(label="bytes / sec", min=0)

Lets break this down a bit...

.es(
  index="metricbeat-*",
  metric="max:system.network.in.bytes",
  split="system.network.name:10",
  kibana=true
)

Query the metricbeat-* index in Elasticsearch for the field system.network.in.bytes, returning values for up to 10 instances of interface names ( split="system.network.name:10" ). The chart should also apply Kibana timescales and filters ( kibana=true ).

.derivative()
Since the value is a counter we need the delta between successive buckets.

.scale_interval(1s)
We want the value expressed as the number of bytes per second.

.if(operator="lt", if=0, then=0)
If the value is less than zero, such as when the counter wraps, use zero instead. This will fix your negative spike issue.

.trim(start=2,end=1)
By removing the first two and last values we can remove a visual artifact that can occur due to partial time buckets at the beginning and end of the chart.

.label(regex="^.* system.network.name:(.+) > .*$", label="$1")
Extract the interface name from the auto-generated label and use that in the legend.

.lines(width=2)
I like a thinner line than the default, especially when displaying multiple interfaces on the same chart.

`.yaxis(label="bytes / sec", min=0)``
Finally, add a label to the y-axis to display the indicated units.

The resulting chart will be similar to this... (this config is a little different than above)

Rob


Timeline - how constract network I/O graph
Metricbeat to show network packets over time
(revolucio) #5

Hi Robert,

I have no words for your reply, really useful and well explained.

I see that my original idea of draw this graph in a line visualisation is not as easy as it seems to me at the beginning.

Thank you, thank you very much....


(Sjaak) #6

The new time series was touted as being able to make this easy but I'm running into the same issues as you. Derivatives produce incorrect results. How you can make a negative graph out of values that are always 0 or higher is beyond me.

In timelion I did it like this. Note that the mvavg requires a additional plugin. I needed this to smooth out my results for my netflow. Might not me necessary using metricbeat depending on how the data is collected.

$src_query='host:1.1.1.1, .es($src_query,metric='sum:netflow.in_bytes').mvavg(1m).scale_interval(1s).divide(1024).label('Up - KBps')

I made different queries for dst, total and also some separate subnets (host:1.1.1.1 AND src_add: 192.168.1.*) so I have 5 or 6 lines in one graph.

edit: scale.interval causes problems when you try to view data over a long period of time because it will create too many buckets. Use the mvavg mod instead.


(Robert Cowart) #7

@Sjaak01, I was about to reply to your related post with a link to what I posted here, but noticed that you found it already. I will also try out mvavg() myself. Thanks for the tip.


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.