Network bandwidth from firewall logs in timelion

[jfs1 (Jean-François Stenuit)]

I configured timelion to graph network usage (in Mbps) for all HTTPS traffic. The timelion expression is :

.es(q="observer.ingress.interface.name:Untrust-4 and destination.port:443",index="fw-*",timefield="@timestamp",metric="sum:destination.bytes").divide(125000).scale_interval(1s).label("Mbps IN"),
.es(q="observer.ingress.interface.name:Untrust-4 and destination.port:443",index="fw-*",timefield="@timestamp",metric="sum:source.bytes").divide(125000).scale_interval(1s).label("Mbps OUT")

The "divide" is taking the bytes from the firewall, multiply by 8 (to get bits) and divides by 1000000 (to get the Megabits).

The graph looks nice, but the scale is completely off. Like : 80Gbps of bandwidth used where the size of the physical links is way less than that.

What can go wrong ?

[stephenb (Stephen Brown)]

HI @jfs1

Most the time Bytes In / Bytes Out are recorded as monotonically increasing counters so you would have to do a derivative to get a rate. Perhaps this is the issue. I am not a timelion expert so I can't help you there.

However both Lens and TSVB have a Counter Rate aggregations / derivatives built in to perform this operations perhaps take a look at them. Lens is a bit easier but TSVB provides more flexibility.

Here is an example in TSVB complete with scaling of the Bytes etc.

You can also apply filters in the Filter section with KQL syntax

Minor Oops. The Label I put Bytes In but I used the metric bytes.out I am sure you will be more careful

[jfs1 (Jean-François Stenuit)]

Hi @stephenb ,

In this case, my stats are coming from firewall logs, so the bytes in/out are for each TCP session and not monotonically increasing.

I keep on investigating.