How to visualize network bandwidth

Elastic Stack Kibana
1

Hi all,

I am using collectd and logstash to push network interface statistics to elastic search and want to visualize actual network bandwidth. The problem is the data sent to elastic search is cumulative data, like total received/sent packets (rx and tx) on the network interface. I wonder how i can de-cumalative the data? The easiest way would be get the diff of the value between two event and divide by the time difference between two log entries.
For example: log entry 1 happens at time t1 with tx1 and rx1. log entry 2 happens at time t2, with tx2, rx2. So the uplink speed would be (tx2-tx1)/(t2-t1). I wonder i i can visualize the uplink speed over time?
Thank you all very much!

BR/Keven

2

As far as I know , thats not possible.
You'll have to calculate the differences when loading the data... Thats unfortunately not where your problems end, because I suspect you'll then end up with number of bytes sent for time period and not a bitrate (as bandwidth is usually described)

Kibana makes things even more difficult because it will/can automatically scale the interval of the graphs, to accommodate small to very large datasets.

If you're happy plotting the "sum of bytes" , then you're gold if you can precalculate the differences . If you know there will always be an entry for every interface for every timeperiod - then you can probably get away with calculating bitrates and plotting average of that.

I ended up having to to code some custom stuff into the kibana backend to plot my bandwidth graphs.

3

Btw, Timelion plugin for Kibana was just released, which makes doing this type of math on time series trivial: https://www.elastic.co/blog/timelion-timeline

4

@Keven_Wang, I think you want the derivative elasticsearch aggregation but not supported by kibana yet.

@tbragin, can you or the team share any rough estimates or milestones about when counter metrics will be supported? Is the plan to be able to include timelion graphs as visualizations in kibana dashboards?

5

Trying to get into timelion specifically to test derivative aggregation, only I seem to have an issue validating my timelion config in the tuturial, what fora would be right to dicuss timelion, here in Kibana I assume?

Appreciate any good timelion doc pointers!

TIA

6

Hi Keven! Im with same problem, Im sending bandwith snmp information to elasticsearch, but is the cumulative data, from ifHCInOctets and ifHCOutOctets oids. Did you solved it ?
Diego

7

I went for derivative aggregation in grafana on top of our ES cluster, also find it's metric dashboards nice(r) :slight_smile:

8

Thanks, Im trying to do it, but I dont understend how.
something like this ?

Pasted image1064×487 29 KB

I send the snmp info collected to logstash and then to elastic.

Thanks !
Diego P.

9

CleverTap
might also help you to understand howto use ES derivate aggregation You need a parent metric/aggregation (not disaplyed) onto which you can make the derivate metric.

Not sure if I needed to enable inline scripts in my ES cluster with this in elasticsearch.yml:

script.inline: true

or it was for some other feature this was needed.

10

Thanks, I must read and study more about that, but I test this code with sense and get what i need, now i have to solve how graph it with kibana and json imput.

{
"aggs": {
"1": {
"date_histogram": {
"field": "@timestamp",
"interval": "5m"
},
"aggs": {
"rx_avg": {
"avg": {
"field": "rx"
}
},
"rx_deriv": {
"derivative": {
"buckets_path": "rx_avg"
} } } } } }

11

Yes if you prefer Kibana for this, I thought that you, like I graphed, with grafana as your SD seems to show and as CleverTab can guide you to. Can't help on Kibana with this :slight_smile:

12

Maybe Kibana wouldn't just yet, add your voice to this issue

Also see this ES blog post

13

Sample of how one of my collectd counter metrics sampled every 300 sec is graphed in Grafana +2.6 through an ES derivate aggregation , again belive you need to enable inline scripting in your ES cluster for this to work

grafana-derivate-graph.png1172×912 97.6 KB

14

Great Steffen !
I make it work.

Pasted image1642×562 63.9 KB

Thanks a lot !

I dont know why if i zoom more than last 24hs the graph show up and dissapear. We will see about next time.
Thanks again
diego

15

Good, remember you'll have to divide your metric with your sampling interval to get per-sec or per-minute whatever you want (_value/300 for our 5 min interval gives us per-sec values) and then you may want set proper Y-axe Unit under Axes tab to other than 'short' to display it nicely :slight_smile:

I've also seen some issues with not-showing graphs initially, but then I do something that'll make the panel refresh. alter a template var, change time zoom a bit... believe it's another issue in Grafana, maybe connected with use of the ES pipeline aggregation, dunno. But at least you have a chance to see derivates instead of counters.

HInt: If you want to trunk of counter reset, which'll give large 'negative' spikes, limit the Y axe value to zero.

16

could you share your solution code-wise ?

17

Hi, in the screenshot u can see what I did. The query.
Diego