# Kafka Consumer cluster that can read dynamic topics and bulk index the data **URL:** \ **Category:** Elasticsearch\ **Created:** [September 15, 2015, 6:18am UTC](https://discuss.elastic.co/t/kafka-consumer-cluster-that-can-read-dynamic-topics-and-bulk-index-the-data/29295 "2015-09-15T06:18:52Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Ramky](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@Ramky](https://discuss.elastic.co/u/Ramky)\ **Post date:** [September 15, 2015, 6:18am UTC](https://discuss.elastic.co/t/kafka-consumer-cluster-that-can-read-dynamic-topics-and-bulk-index-the-data/29295/1 "2015-09-15T06:18:52Z")
I need to build scalable Kafka consumer cluster, which reads data from dynamically added Kafka topics and bulk index the data to Elastic-search. At present,i developed a java consumer client jar which reads data from given list of Kafka topics and feed them to ES cluster using bulk indexing. I also tried using Spark, but performance is 60% lower than running jar from command line. I will have new topics added to Kafka cluster on fly. So i need to develop a scalable cluster for consuming various topics of Kafka data and feed the data to ES. Please suggest me the better way to build a scalable Kafka Consumer and bulk index the data. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 15, 2015, 6:25am UTC](https://discuss.elastic.co/t/kafka-consumer-cluster-that-can-read-dynamic-topics-and-bulk-index-the-data/29295/2 "2015-09-15T06:25:23Z")
Instead of building that jar you could have just used Logstash - [https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html) Not sure what you can do around the dynamic aspect. ---
**Author:** ![tinle](https://avatars.discourse-cdn.com/v4/letter/t/c77e96/32.png) [@tinle](https://discuss.elastic.co/u/tinle)\ **Post date:** [September 15, 2015, 5:23pm UTC](https://discuss.elastic.co/t/kafka-consumer-cluster-that-can-read-dynamic-topics-and-bulk-index-the-data/29295/3 "2015-09-15T17:23:02Z")
Kafka 0.8.2 or newer supports whitelist and blacklist regex for topics. That is most likely what you are looking for. I use KCC (kafka-console-consumer) on command and pipe input to ingest Kafka data. KCC supports --whitelist and --blacklist flags. ---
**Author:** ![Ramky](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@Ramky](https://discuss.elastic.co/u/Ramky)\ **Post date:** [September 16, 2015, 10:00am UTC](https://discuss.elastic.co/t/kafka-consumer-cluster-that-can-read-dynamic-topics-and-bulk-index-the-data/29295/4 "2015-09-16T10:00:30Z")
Tinle, Thank you for your response. I will try your suggestion. ---
**Author:** ![Ramky](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@Ramky](https://discuss.elastic.co/u/Ramky)\ **Post date:** [September 16, 2015, 10:01am UTC](https://discuss.elastic.co/t/kafka-consumer-cluster-that-can-read-dynamic-topics-and-bulk-index-the-data/29295/5 "2015-09-16T10:01:48Z")
Mark, Thanks for your suggestion. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:50pm UTC](https://discuss.elastic.co/t/kafka-consumer-cluster-that-can-read-dynamic-topics-and-bulk-index-the-data/29295/6 "2017-07-05T23:50:04Z")