Pattern filter in filebeat or ingest node?

Elastic Stack Beats
1

Hello,

I have a setup with a filebeat agent that sends messages to an elastic cluster.
I need to filter messages that goes to the cluster, and I have 2 options:

  • Using pattern filters on filebeat
  • Using filtering pipelines on ingest node

I have a lot of data (approx 500GB per day, and only 1% will be stored in elastic nodes).
Is there any recommendations on what is the most confortable solution?

Thanks!

2

Hi @rverchere and welcome :slight_smile:

I have a lot of data (approx 500GB per day, and only 1% will be stored in elastic nodes).
Is there any recommendations on what is the most confortable solution?

It depends, but quite probably the best option is to filter them out already in filebeat, this way you avoid the network traffic caused by these messages that you are going to drop in any case.

To drop events from filebeat, you can use the drop_events processor.

3

Hey, thanks for your feedback!

I think I will go for your solution, and if I have time or find something that does not fit my needs, I will go with pipelines on ingest nodes.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.