I believe Elasticsearch supports Spark Structured Streaming since version 6. I have seen examples of WRITING structured streaming data from spark to Elasticsearch in append mode. But I have not seen a single example of reading from Elasticsearch.
For example, here is how you write to ES:
query = df.writeStream \
.outputMode("append") \
.queryName("writing_to_es") \
.format("org.elasticsearch.spark.sql") \
.option("checkpointLocation", "/tmp/") \
.option("es.resource", "index/type") \
.option("es.nodes", "localhost") \
.start()
query.awaitTermination()
But how do you read from ES using spark structured streaming ? Please share an example .....
Thanks,
Saurab