Hi all, very new to ES and i started writing my own logging function writing direct to Elasticsearch with C# to log network packet counters on a minute basis (IP pairs, bytes in/out), and run a query by sum over multiple daily time indices, this would be for bandwidth accounting (to run on a specific day, and get the data-in and data-out bandwidth for the past calendar month for a specific IP).
I then saw Logstash with Netflow input which would be a lot more appropriate and would give details such as protocol and countries and a Kabana dash to search with, I can get that setup, but is it possible to write an integration to query the data store (es?) for the monthly bandwidth as above, or some form of filter to output daily bandwidth in/out per IP to a file or database?
Ideally i would like my external app to query on a scheduled basis and pull out this data which would form part of the billing system.
Are there specific schema or API that i could start with?
Thanks for the reply Magnus, sorry if i came across wrong, what i meant is to use logstash and one of the pre-built plugins that will accept netflow data, flow in to logstash and get pumped out to elasticsearch.
From there logstash will write to elasticsearch, all using the pre-existing solution stack.
I would then have my custom app query es via the NEST C# API (or rest etc), wondering if there is a specific schema for the netflow data that is stored from logstash, wondering if there is any documentation that could indicate what it is currently, and what changes would be in future. Once i know the schema, i could then write the native es queries accordingly.
This way, i could use kabana for visual ad-hoc queries, and the custom app to generate bandwidth statistics, all using the same es data store.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.