Capture bandwidth of each unique url

[Allan_John]

Hi,

I have some urls that come in this format in elastic search:

192.168.1.1/livestream/test/media.m3u8
192.168.1.1/livestream/test/media_aqbcd00100.ts
192.168.1.1/livestream/test/media_aqbcd00101.ts
192.168.1.1/livestream/test/media_aqbcd00102.ts
192.168.1.1/livestream/test/media_aqbcd00103.ts
192.168.1.1/livestream/test/media_aqbcd00104.ts

I need to extract the urls in such a way that I need to make it unique. For example I need to get like this:
192.168.1.1/livestream/test

The purpose is to calculate the total bandwidth transferred from the server, for each unique url. So in this case i can calculate the total bandwidth transferred for "test" consumed by various IP's and the bandwidth consued by each url , in this case : "192.168.1.1/livestream/test"

Is it possible in Scripted language in Kibana?

Thanks,

[cjcenizal]

Hi Allan,

Yes, I think you can use scripted fields to do what you're trying to do. You should be able to create a scripted field that extracts a substring from your URL string. Then you can create a Terms bucket aggregation on that scripted field, and configure your metric aggregation to sum your bandwidth field.

For example, you could create a scripted field called baseUrl that does this:

def text = doc['url'].value;
int firstIndex = text.lastIndexOf('/');
int lastIndex = text.length();
return text.substring(firstIndex, lastIndex);

This scripted field will return a value of "192.168.1.1/livestream/test/" for the example URLs you shared.

Then, assuming your bandwidth is represented by a field called bytes, you'd configure your visualization like this:

This will show you a visualization of the top 5 baseUrls which consume the most bandwidth.

Please let me know if this helps!

Thanks,
CJ

[cjcenizal]

BTW, here's the documentation of other methods you can use to manipulate String fields: https://docs.oracle.com/javase/7/docs/api/java/lang/String.html. Might come in handy if you want to try something else other than lastIndexOf and substring.

CJ

[Christian_Dahlqvist]

Even though it appears possible to do this using scripted fields in Kibana, the most efficient and scalable way to do this would probably be to instead extract the url you want to aggregate on at indexing time and store it in a separate field mapped as a keyword.

[Allan_John]

Hi CJ,

Thanks for the reply.

What if I need to extract the text "test" alone. Getting the lastIndexOf
the output in the above command will work?

Thanks,

[cjcenizal]

Hi Allan, have you taken a look at the docs I linked to? You can pass additionally arguments to lastIndexOf to "walk" backwards from one slash to the slash preceding it. Then you can continue to pass these indices to substring to extract out the different parts of the URL. Though you might want to consider Christian's suggestion if you're planning on building a long-term (i.e. scalable) solution.

Thanks,
CJ