Previously, I was using storm topology to build my pipeline to suffice my
use case which was - Reading from Kafka topic through Kafka spout, process
these tuples in Bolt, then emit them to ElasticSearch bolt to index my
tuples. In storm topology, I was able to acknowledge each tuple i was
indexing and these acknowledgments were sent from EsBolt to spout and
finally to Zookeeper.
Later I realised that Storm is not the appropriate component for my use
case and I switched to Spark. I am doing the same thing with Spark -
Reading from Kafka topic, reading messages in Spark stream, pass this
stream to a component which converts this stream to RDD and finally indexes
these RDDs. But now, there is no mechanism for acknowledgements. But I
still want to send acknowledgements for each messages being indexed to the
previous component.
My code looks like this -
JavaEsSpark.saveJsonToEs(jrd, "index_{date}/type");
How can I get acknowledgements in Spark-ElasticSearch framework?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f73a57fd-1f5a-41f3-b6ec-077961668687%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.