I'm new to packetbeat, and dived straight into exploring packetbeat events in Kibana setup to listen to a small preproduction environment. And I noticed a reoccuring spike every 10 seconds of 300 events being generated (the other 9 seconds have <20 events each second). The events start with http connections to my ES (initiated by packetbeat itself) and then a lot of flow events.
I don't understand flow events very well, other than they're bidirectional comm channels. Are these flow events being generated by packetbeat after an HTTP connection is established? If so, it seems like good practice to exclude these 300 events every 10 seconds from being reported as they drown out the other network traffic analysis I'd like to do.
Anyone else see this traffic heuristic from packetbeat?
I see. I have added port 80 to my list of ports under HTTP, and because I'm using AWS ElasticSearch which uses port 80, I'm seeing beatpacket traffic being logged in ElasticSearch.
yes ok then that makes sense, there is an option to packetbeat to exclude "source" data. IE data that is originating from the server.
I am not sure if you can change ports on AWS but you could choose a different transport protocol. to an intermediate service and then forwarding that.... something like
packetbeat -> logstash (beats-iinput) on a different server -> elasticsearch
this way you capture what you want, without any special configuration, Logstash will act as a small buffer for elasticsearch. Of course you can go more complex then that. But this would be a simple win to accomplish what you want.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.