I'm getting error
"EsHadoopIllegalArgumentException: Cannot detect ES version-typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'*
I'm using pyspark to write my dataframe to elasticsearch cluster like this:
df1.write.format("org.elasticsearch.spark.sql")\ .option("es.nodes", host)\ .option("es.port", port)\ .option("es.net.http.auth.user", username)\ .option("es.net.http.auth.pass", password)\ .option("es.resource", indexName)\ .option("es.net.ssl.keystore.location", pathToCAFile))\ .mode('overwrite')\ .save()
I've tried wan only options but they make no difference.
I've checked my cluster connectivity using curl, it's working totally fine and I'm able to connect to elasticsearch server using python also but Pyspark is giving me hardtime here. I'm using the same version of EsHadoop-jar file also as per my elasticsearch cluster version.
Please help me with this issue.