Is it good practice to exclude Packetbeat traffic?

[jnmoore (Jerroyd Moore)]

Greetings all,

I'm new to packetbeat, and dived straight into exploring packetbeat events in Kibana setup to listen to a small preproduction environment. And I noticed a reoccuring spike every 10 seconds of 300 events being generated (the other 9 seconds have <20 events each second). The events start with http connections to my ES (initiated by packetbeat itself) and then a lot of flow events.

I don't understand flow events very well, other than they're bidirectional comm channels. Are these flow events being generated by packetbeat after an HTTP connection is established? If so, it seems like good practice to exclude these 300 events every 10 seconds from being reported as they drown out the other network traffic analysis I'd like to do.

Anyone else see this traffic heuristic from packetbeat?

[eperry (Ed)]

Ok packet beat will listen on any port you have configured to listen on for HTTP

I presume your using the Elasticserh output for packetbeat, which should by default be 9200.

Just make sure that port is not in your list of ports and you will not capture the data.

If that does not make sense please provide your config and maybe even example data of what you are seeing

[jnmoore (Jerroyd Moore)]

I see. I have added port 80 to my list of ports under HTTP, and because I'm using AWS ElasticSearch which uses port 80, I'm seeing beatpacket traffic being logged in ElasticSearch.

[eperry (Ed)]

yes ok then that makes sense, there is an option to packetbeat to exclude "source" data. IE data that is originating from the server.

I am not sure if you can change ports on AWS but you could choose a different transport protocol. to an intermediate service and then forwarding that.... something like

packetbeat -> logstash (beats-iinput) on a different server -> elasticsearch

this way you capture what you want, without any special configuration, Logstash will act as a small buffer for elasticsearch. Of course you can go more complex then that. But this would be a simple win to accomplish what you want.