Logstash Bandwidth Accounting


(Chris Morley) #1

Hi all, very new to ES and i started writing my own logging function writing direct to Elasticsearch with C# to log network packet counters on a minute basis (IP pairs, bytes in/out), and run a query by sum over multiple daily time indices, this would be for bandwidth accounting (to run on a specific day, and get the data-in and data-out bandwidth for the past calendar month for a specific IP).

I then saw Logstash with Netflow input which would be a lot more appropriate and would give details such as protocol and countries and a Kabana dash to search with, I can get that setup, but is it possible to write an integration to query the data store (es?) for the monthly bandwidth as above, or some form of filter to output daily bandwidth in/out per IP to a file or database?

Ideally i would like my external app to query on a scheduled basis and pull out this data which would form part of the billing system.

Are there specific schema or API that i could start with?


(Magnus Bäck) #2

is it possible to write an integration to query the data store (es?)

Sure. Kibana is just a visualization layer on top of ES.

Are there specific schema or API that i could start with?

The schema is up to you; ES has few opinions there. You can find available APIs in the ES documentation.


(Chris Morley) #3

Thanks for the reply Magnus, sorry if i came across wrong, what i meant is to use logstash and one of the pre-built plugins that will accept netflow data, flow in to logstash and get pumped out to elasticsearch.

From there logstash will write to elasticsearch, all using the pre-existing solution stack.

I would then have my custom app query es via the NEST C# API (or rest etc), wondering if there is a specific schema for the netflow data that is stored from logstash, wondering if there is any documentation that could indicate what it is currently, and what changes would be in future. Once i know the schema, i could then write the native es queries accordingly.

This way, i could use kabana for visual ad-hoc queries, and the custom app to generate bandwidth statistics, all using the same es data store.

Pointers much appreciated.


(Magnus Bäck) #4

Oh, I see. I've never used the netflow plugins so I have no idea what the data will look like. I suggest you experiment.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.