Sum network traffic



Since this issue has been closed by @jsoriano I'm asking for a solution here.
I was just trying out Metricbeat to see if I can log outgoing traffic in an LXC container. I couldn't specify other option than the interface (e.g., ports) but I let that go. Unfortunately, I couldn't tell the outgoing traffic in a specific time period after enabling the system network metric. I don't need visualizing it, I simply need a sum.
The hard part is, that whenever the container resets, the counter resets. This isn't, of course, Metricbeat's fault, but there isn't any way to query the data from ES where I can say that I need all the "max" values between X and Y, except for the first one, where you need to take the first document and subtract its
value from the first "max".
I was hoping that there will be an optional setting for non-incremental values for those who need this.
I played around a bit with derivative queries but I couldn't get the result I needed.
My question would be if I'm missing something or this is simply impossible with Metricbeats?


(Kaiyan Sheng) #2

Hello @YvorL! I believe we can do sum in Kibana when creating a Visual Builder in visualization.


Hi @Kaiyan_Sheng!
That may be true, but I need an ES query solution since I need to forward the response to an application on a container basis (speaking of hundreds of containers). So, unfortunately, visualizing it won't help my case. :frowning_face:

(ruflin) #4

Not sure I fully understood yet what you need. So the values are counters. If you want to know to total network traffic between point a and b you are interested int he value b-a?


The values are counters, but every time a container restarts (which happens randomly) the counter restarts from 0. So if I'd like to know the SUM of a 15-day interval where the virtual network card's traffic has been restarted 3 times (4th, 9th, 14th), I'd have 4 maximums (given that I can extract those):

  • 4th day maximum (A)
  • 9th day maximum (B)
  • 14th day maximum (C)
  • 15th day maximum (D)

Where I'd need to subtract the first day's first document's value from A, to get the first subperiod's data. then add B and C maximums, and lastly, subtract the first document after C maximum from D to get the last subperiod. After that, I'd get the total for that 15 days. This is for only one container, for a smaller time period and for the example's sake only 3 restarts at given intervals. Unfortunately, the reality is far more complicated :frowning:
If the values in the documents were non-incremental, it'd be sooooo easy.