org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: org.elasticsearch.spark.sql. Make sure the provider name is correct and the package is properly registered and compatible with your Spark version

Elastic Stack Elasticsearch
,
1

Hi Everyone,

Currently I want to write pyspark dataframe to Elasticcloud index

Below is the code, however pyspark is unable to find the org.elasticsearch.spark.sql.

Environment - Databricks
Databricks Run time - 16.3

(   
    df
    .write
    .mode('overwrite')
    .format("org.elasticsearch.spark.sql")
    .option("es.resource", "INDEX_NAME")
    .option("es.write.operation", "index")
    .option("es.net.http.header.Authorization", "TOKEN")
    .option("es.nodes", conf['host'])
    .option("es.port", conf['port'])
    .save()
    )

Error

 org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: org.elasticsearch.spark.sql. Make sure the provider name is correct and the package is properly registered and compatible with your Spark version

Please provide you suggestion.