Sum of source bytes seems impossibly large

[willemdh]

Hello,

Was browsing through Kibana SIEM on 7.5.2 and discovered some weird 'Bytes In', 'Bytes Out' metrics. After investigating, it seemd like some servers were sending huge amounts of traffic to my Elastic ingest nodes. I'm talking about 45 TB / 24 hours to each ingest node...

After some investigation, this data came from the flow packetbeat module:

packetbeat.flows:
  timeout: 30s
  period: 10s

So what is going on here? How does the Packetbeat flow functionality calculate source.bytes? The result is that Kibana network SIEM shows very weird results.. Is this a known issue?

Grtz

Willem

[willemdh]

Anyone?

[willemdh]

Same issue on 7.6.1 by the way.. Check this lol:

This is on last 24 hours for 1 server. 1,7PB???? Unrealistic...

[Mike_Paquette]

Hi @willemdh, Yes, those are some really big numbers! This could be because packetbeat reports bytes in a flow as a cumulative counts since the flow began, rather than as an incremental count since the last event. So when you sum them up, you get some huge values, the longer the flow lives, the sum (bytes) grows unexpectedly.

One option is to set packetbeat.flows.period: -1s so that you receive events only at the end of the flow. The byte values will then be accurate, but of course, you won't have any bytes shown for longstanding flows that have not closed yet.

Please let us know if this helps.

[willemdh]

@Mike_Paquette Thanks for the explanation. I will try your suggestion. It seems that the documentation indeed mentions this:

So maybe the SIEM datatable containing Bytes In / Bytes Out could be tuned so it only lists the sum of traffic where flow.final equals true? That way we can still use intermediate flow reports without having this weird side effect in SIEM.

Grtz

Willem

[willemdh]

Setting period to -1 works. So what about:

So maybe the SIEM datatable containing Bytes In / Bytes Out could be tuned so it only lists the sum of traffic where flow.final equals true? That way we can still use intermediate flow reports without having this weird side effect in SIEM.

[Mike_Paquette]

Thanks for the reminder @willemdh. Yes, I'll create an issue for tuning the queries associated with the Source IP and Destination IP tables in the Hosts views as you suggest. We should base this on ECS fields, of course