Can we connect esrally to KAFKA

Hi All,

I am exploring a pipeline esrally->Kafka->ELK for one of the POC.
i understand, esrally works fine ( and only ) with ElasticSearch. Is there any way, where i can use esrally as load generator and feed that data to kafka ?

Hi,

you're right that Rally is intended to work with Elasticsearch. While its core is agnostic to the system being benchmarked (as long as it uses a request-response interaction model) and it might be doable to make it work with Kafka, it is a bit of a stretch and requires a some trickery and knowledge of Rally.

For starters you definitely need to write a custom Rally runner and configure the Kafka client yourself there. To provide the data you want to send to Kafka, you'll likely need a custom parameter source.

There are maybe a few more details that need to be considered but if I were to attempt it, I'd start there. I am not familiar with benchmarking tools for Kafka but I wonder whether it would be easier to go down that path.

Daniel

Thanks Daniel. I completely agree with you.
I am looking for some pointers, where i can either reuse or modify workload for producer -> Kafka -> ELK pipeline.

Do we have any github project or any blog for similar workload ?