Input Filtering on fields or bpf filter before sending to elastic

Elastic Stack Beats
1

Hi,

I would like to known if it's possible to filter data in packetbeat, based on protocol fields.
For example, I would like to exclude some ips from client_ip or host from http.request_headers.host.

Also, is it possible to configure bpf filters on interfaces ?

2

For now the short answer is 'no' to both questions.

We want to have filtering in beats, but we're still looking for the best way to do it.

There is an implicit BPF filter set by Packetbeat (computed from the ports configured in the configuration file) but there's currently no way to explicitly set it. Adding this should be fairly easy, we'd welcome a pull request for it if you want to play with it.

3

Thanks for the reply,
I was trying bpf_filter: and didn't understand why it was not working.

I'll try to have a look at it, it should be easy to define a custom bpf_filter and add something like
bpf_filter += "and port ...."

4

I'm never code with go language, but maybe a simple

func (sniffer *SnifferSetup) Init(test_mode bool, events chan common.MapStr) error {

  • config.ConfigSingleton.Interfaces.Bpf_filter = tcp.BpfFilter()
  • config.ConfigSingleton.Interfaces.Bpf_filter += tcp.BpfFilter()

could work ?

5

I did something like this as part of this pull request, more precisely here. However, instead of attempting to combine the two filters, I just made the user supplied one overwrite the existing one. This is because combining them is easy with simple examples but gets complex with VLANs, multiple conditions, etc.

6

I like the idea of being able to filter too. One use case could be to actually store request values depending on response code. I don't have to store response body or request parameter (is there a way to actually save request body of POST request as well? Haven't figured it out yet) for a request returning 200 code, but I want to know what the complete request is in case of failure. So filtering depending on response code would be awesome!

7

We agree this is important and we'll be adding a flexible way of filtering to support use cases like this. But I don't have an ETA yet.

8

You can follow the status of this feature request in Github: https://github.com/elastic/libbeat/issues/111