Load test log stash through file beat

Elastic Stack Beats
1

I would like to stress test our logstash environment, through filebeat, the idea is to generate log files
with different sizes intervals and call filebeat programmatically to send logs to log stash, any ideas?
How people stress test their log stash environment?

Thanks,

Kasi

2

If you want to do the above with Filebeat, it is probably best to keep Filebeat running on only move the files that should be read into the directories that have to be crawled at the right time. This seems easier then to "remote" trigger Filebeat.

In general for Logstash load testing, it is probably best to use the Logstash forum.

3

While you can use filebeat, when do this what you're testing is:

disk IO (IO cache) -> filebeat -> logstash output -> network layer -> beats input -> logstash pipeline -> filters -> ...

You want to test logstash only, or your complete processing pipeline? For testing beats -> logstash (in case I don't want full filebeat), I use https://github.com/urso/generatorbeat.

Logstash itself has a generator plugin, plus file and event stdin input plugins that can be used to remove some components.

Testing with filebeat I normally use pre-made files + a shell script deleting the old registry file and restart filebeat to ensure the complete file is send again. This is a pretty static setup. Your approach with creating files/logs dynamically you can achieve by having filebeat running in background and just write new files into the directories watched by filebeat. Alternatively you can use pipes if you configure filebeat to use the stdin prospector.

4

Hello Steffen,

I want to stress test my entire stack and found this discussion. Think generatorbeat can help me.

Node (with generatorbeat) > HAProxy -> 2 LS shippers -> Redis Queue -> 2 LS Indexers -> ES cluster -> Kibana

I managed to install the generatorbeat but I need some help with the configuration.
Currently I have 2 files under the /etc folder: beat.yml and fields.yml

Where should i put my logstash output config? Can you point me to any documentation that could help me?

Kind Regards,

Stefaan

5

normally we build configurations via make update. This combines (a simple cat) etc/beat.yml with output settings in libbeat/_meta/config.yml. => update beat.yml