I wanted to figure out where all my bandwidth was going at home. I set up a switch and a sniffer port and a friend recommended Packetbeat, Elasticsearch, and Kibana. So far, it seems very interesting, but ... I am not sure of the results I'm getting.
In my "top hosts creating traffic" graph, I have local IP's along with remote ones. I figure most of my bandwidth is spent on Netflix and YouTube, so I suspect that most traffic is heavily download biased. And yet, my local IP's end up in the top talkers pretty regularly. Does that mean I'm actually pushing gigs and gigs out to the Internet? That doesn't seem likely. Particularly from the IP's it lists.
One of my top talkers is 255.255.255.255. A friend put in "sources by destination," and I'm not sure that's the right search. So, his theory is that all of the destination traffic is searched and then bucketed by unique IPs. That's what I want. But it looks to me more like a search of top IP talkers and then how much they send to 255.255.255.255. Clearly, there is too much broadcast traffic. What's the best way to figure out what's causing it?
I'd love to have an aggregate graph because I feel that would tell the truth of things. How can I get a list of IP's and then a bar graph of when they are source.ip with source.stats.net_bytes_total and dest.stats.net_bytes_total? That should filter out all the WAN IP's as the top talker/receivers should all be local IP's.
Or is there a different way to bucket everything according to the IP address? (MAC works, too.)
P.S. If someone can point me to a good tutorial / class / book on how to understand the query language, I'd appreciate it. Just tinkering with various JSON's is not satisfactory.
Glad you're looking into PacketBeat and the Elastic Stack to figure out what's up with your home network
Just making sure you've got the bases covered: ElasticSearch can receive and parse what you're sending it (with Packetbeat or otherwise) without any prep at all. However if you want a better getting started experience, there are pre-made dashboards you can import for Packetbeat.
So make sure you go through the full PacketBeat Getting Started. Step 3* and Step 4 are the ones that will ensure you have ES and Kibana fully set up for looking at your network traffic.
If you hadn't done these steps and you've done them now, you'll want to delete the daily indexes that have been created from the "Management" section in Kibana: Management/ElasticSearch/Index Management, and start packetbeat again. See the note at the bottom for more details.
With the Getting Started done, you will find pre-made dashboards in Kibana. Go to the "Dashboard" section on the left, and you should see a list of dashboards:
Start with "Packetbeat Overview" and "Packetbeat Flows".
Every square section of these dashboards are made up of visualizations which you can edit in the "Visualization" section. Of course you can go in there and look how they're built, then create your own. For a tutorial in building your own visualizations, you can look at this tutorial and focusing on the "Basic Visualizations", "Data" only. The other sections require deeper ElasticSearch knowledge.
If we continue working backwards, the "Discover" section is where you can see your full stream of raw monitoring events. In there you can look at every field that make up your events. When you go in there, click "Open" at the top, and load the "Packetbeat Search" saved search. It'll give you an idea of everything flowing through the system. Hit the small triangle on any line in the table below to see an event in all it's detail
Good luck in your exploration
Mathieu
Step 3 -- the index template -- is an ElasticSearch concept similar to creating a DB table schema. Without them, ElasticSearch can detect lots of types without any preparation. But by installing the proper index template you'll have an even better experience. For example your IP addresses will be of type ip, and you'll be able to search for CIDR matches & so on. If you hadn't installed the index template, you'll want to delete your packetbeat indexes, to have the proper types from the index template be applied.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
