the metric filter averages over time. One has to wait for quite some time until these values are more stable to get some better idea of throughput.
There are a many factors regarding filebeat performance. Just sending files to /dev/null on physical machine I was able to process like 95k eps. filebeat throughput depends on disk IO (unless files still buffered by OS caches) and downstream performance. E.g. if sending directly to elasticsearch indexing performance in elasticsearch. If sending to logstash throughput depends on processing time within logstash + performance even more downstream. This is due the outputs generating back-pressure if they can not keep up slowing down event generation in filebeat (as we don't want to drop any events).
See related post for measuring throughpt directly from filebeat: Filebeat sending data to Logstash seems too slow
To get a first idea on performance i'd run these tests:
- have filebeat publish to console and redirect to /dev/null. Alternatively configure file output in filebeat
- have filebeat->logstash, but configure logstash stdout plugin only with 'dots' codec. Use
pvto measure throughput.
Advantages if 'pv' over metrics plugin is:
- no additional filters/computations in logstash
- more instantaneous rate