Testing and benchmarking log collection pipeline

Elastic Stack Logstash
1

I'm looking for tools, or a set of tools, to test and benchmark logstash+kafka+ES setups :

  • define test cases (log types, throughputs)
  • maintain a base of log examples
  • generate inputs for logstash at a given rate from the example files (some like https://www.snmpsoft.com/cmd-tools/syslog-gen/)
  • reproduce results with minimal humain vérification

Does something like a standard toolset exist to do thing sort of thing ?

2

I think Elastic have a bunch of performance test suites somewhere, but other than that I haven't heard of the kind of end-to-end testing tools that you describe. However, for testing Logstash filters specifically I think my own tool Logstash Filter Verifier is useful. Verifying that filters are working as expected is probably easier done on the unit level than on the system level.

3

Thanks Magnus,

I agree with you on unittesting your filters and patterns. I actually build
a tool similar to yours (a lot simpler, using rspec) that allows you to
define test cases for patterns, as well as filters: Logstash Tester
https://github.com/gaspaio/logstash-tester.

I think an end to end testing suite would be really useful, for those cases
when you have a defined set of logs that gets parsed, indexed and triggers
some some of app behavior, and you want to test the whole chain and
reproduce behaviros. Oh well, i'll probably end up hacking something myself
for this.