I am running a 3-node ELK 6.3 cluster. The use case is simple and generic. A bunch of machines generate logs, Filebeat ships those logs to one Logstash instance (What!! can Logstash handle it? yes, for now, Elasticsearch seems to be the bottleneck) which sends to Elasticsearch.
I have a set of automated scripts that start the benchmarking, runs for a while and the x-pack monitoring dashboard gives me the indexing speed, CPU Utilization %, Memory Usage et al.
I came across Rally today and was intrigued as it was Out-of-box solution for benchmarking Elasticsearch. But, that seems to be very specific to Elasticsearch. I would be happy to just benchmark my Elasticsarch for now, but i need the whole workflow. Logs shipped from filebeat->Logstash->Elasticsearch.
Is this possible to do with Rally? We can create custom tracks in Rally but those seem to have data files in the tracks. This translates to having all logs in the track, but can these custom tracks be used to generate logs, have filebeat, logstash do its thing and we just benchmark the Elasticsearch? If it is possible, i would just call my automated scripts in these tracks which initiates the log generation process.
Also, any reviews on Rally, like how easy it is to use? Issues faced etc would be highly appreciated.