Any suggestions on creating watches for derivative values?

Elastic Stack Elasticsearch
,
1

I'm interested in creating a threshold alert for bits/s or packets/s of my data. This is leading me down a rabbit hole, and it seems like doing any analysis (sorting/aggregation) on this kind of per second data is quite difficult; the only option seems to be to use the TSVB and create a visual with that.

I considered using data transforms to try and do things by "entity" but I can't figure out how to exactly do that given that I need the per second rates for a given source.ip. I'm using data in the ECS format so I have fields like network.bytes or network.packets.

Any suggestions on being able to do more interesting manipulations of data that is per second?

2

Summing up the network.bytes or network.packets over time (using a date_histogram aggregation), gives you an effective rate, because you have a number of bytes per unit time. No need to calculate a derivative, unless you're interested in the rate of change (i.e. bits/sec/sec).

What are you going to set your threshold at? Why not consider using ML to dynamically find anomalies in your network data?

3

Wouldn't the date histogram give me the rate depending on the bucket? So if I'm looking at a month, and the buckets are every 24 hours , then I'm looking at packets/24 hours? I want to look at packets/s no matter the bucket size.

I'm looking at creating a watcher that will tell me when the packets/s passes a threshold. I would be interested to use ml as well, but I'd like to track the per second rate. Does that make sense?

4

Yes, of course the date_histogram will give you whatever the fixed_interval is. You can set it to any value you want.

I need to ask the obvious question, however - what granularity are you getting this raw data in?

5

At least in the current version of Kibana, there's no fixed-interval but rather a minimum interval... but with minimum, it doesn't force the value I put in. (e.g. if I put second as the minimum it doesn't show second buckets, but whatever it thinks makes most sense).

My data is log data that's coming in on a ms scale.

6

Sorry, what I meant when I asked about the granularity of the data is what is the sampling interval of the data? In other words, if you have a network.bytes value for a specific source.ip, what is the typical time in between adjacent samples?

7

Hmm it's hard to say, most of the time there are at least 2, sometimes up to 10 per second. However sometimes there is 1 event for a source ip or none in a given second.

8

Ok, great - that helps me understand the approximate velocity of the data you're dealing with.

So, my next question is:

Do you really want/need an alert for a 1 second window of time on this data? In other words, if there's a spike of network traffic above your threshold at 12:38:06, but is back in range at 12:38:07, but spikes again at 12:38:08 you really want two alerts? And, you want to do this for every source.ip in your data?

9

So that's a decent question; I feel like people want to know their mbps speed to understand bandwidth usage. If I can do 100mbps for my network, understanding the data usage as a percent of that is important no? Or is this best dealt with one of the beats that have this as an explicit value rather than the logs that mention packets or bits transferred?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.