Hi All,
I am exploring a pipeline esrally->Kafka->ELK for one of the POC.
i understand, esrally works fine ( and only ) with ElasticSearch. Is there any way, where i can use esrally as load generator and feed that data to kafka ?
Hi All,
I am exploring a pipeline esrally->Kafka->ELK for one of the POC.
i understand, esrally works fine ( and only ) with ElasticSearch. Is there any way, where i can use esrally as load generator and feed that data to kafka ?
Hi,
you're right that Rally is intended to work with Elasticsearch. While its core is agnostic to the system being benchmarked (as long as it uses a request-response interaction model) and it might be doable to make it work with Kafka, it is a bit of a stretch and requires a some trickery and knowledge of Rally.
For starters you definitely need to write a custom Rally runner and configure the Kafka client yourself there. To provide the data you want to send to Kafka, you'll likely need a custom parameter source.
There are maybe a few more details that need to be considered but if I were to attempt it, I'd start there. I am not familiar with benchmarking tools for Kafka but I wonder whether it would be easier to go down that path.
Daniel
Thanks Daniel. I completely agree with you.
I am looking for some pointers, where i can either reuse or modify workload for producer -> Kafka -> ELK pipeline.
Do we have any github project or any blog for similar workload ?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.