Ship only a percentage of logs to logstash

Elastic Stack Beats
1

How can I configure filebeat to only ship a percentage of logs (a sample if you will)

In my application's log folder the logs are chunked to about 20 megs each. I want filebeat to ship only 1/300th of that log volume to logstash.

Can filebeat filter a percentage of all log data? Haven't seen anything in the docs about this

2

Filebeat cannot, you may be able to with Logstash and something like the aggregation filter.

1 Like
3

Thats not going to work for me, I need to pare down the log volume before I send it over the wire to logstash.

Can another product like nxlog do this if Filebeat can't? I'd think this scenario would not be uncommon (ship only a sample of logs).

4

In general what I saw is that to reduce the log volume is that people only send error or warning log lines instead of also info, so the critical information still reaches the elasticsearch cluster. So I would recommend you to look a way to reduce the amount of logs based on some criterias instead of just a percentage.

I'm not aware that any of the other products can do that but you must check nxlog and the others in detail to see if they support it.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.