Streaming Sink doesnt work without checkpoint?

Streaming sink with commit log disabled still throws the exception , any reason why it looks for checkpointlocation when i have option set not to use checkpoint ? :frowning:

org.apache.spark.sql.AnalysisException: checkpointLocation must be specified either through option("checkpointLocation", ...) or SparkSession.conf.set("spark.sql.streaming.checkpointLocation", ...);

df.writeStream
.format("org.elasticsearch.spark.sql")
.option("es.resource", "document")
.option("es.mapping.id","id")
.option("es.spark.sql.streaming.sink.log.enabled",false)
.option("es.index.auto.create","no")
.option("es.nodes", "")
.option("es.write.operation", "upsert")
.outputMode("append")
.start("document")

using databricks 5.2 with Spark 2.4.0 , Scala 2.11 and elasticsearch_spark_20_2_11_6_6_1.jar

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.