Previously, I was using storm topology to build my pipeline to suffice my
use case which was - Reading from Kafka topic through Kafka spout, process
these tuples in Bolt, then emit them to ElasticSearch bolt to index my
tuples. In storm topology, I was able to acknowledge each tuple i was
indexing and these acknowledgments were sent from EsBolt to spout and
finally to Zookeeper.
Later I realised that Storm is not the appropriate component for my use
case and I switched to Spark. I am doing the same thing with Spark -
Reading from Kafka topic, reading messages in Spark stream, pass this
stream to a component which converts this stream to RDD and finally indexes
these RDDs. But now, there is no mechanism for acknowledgements. But I
still want to send acknowledgements for each messages being indexed to the
My code looks like this -
How can I get acknowledgements in Spark-ElasticSearch framework?
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f73a57fd-1f5a-41f3-b6ec-077961668687%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.