Attempting to read/write with es-hadoop however I am getting the following error in Databricks
EsHadoopInvalidRequest: [HEAD] on [index_name] failed; server [https://serveraddress.com/es:443] returned [405|Method Not Allowed:]
Note the port comes after the prefix which contradicts what the documentation says here for prefix
es.nodes.path.prefix (default empty)
Prefix to add to all requests made to Elasticsearch. Useful in environments where the cluster is proxied/routed under a certain path. For example, if the cluster is located at
someaddress:someport/custom/path/prefix, one would set
My config in databricks just looks like
spark.read.format("org.elasticsearch.spark.sql") .option("es.nodes", es_url) .option("es.nodes.wan.only","true") .option("es.nodes.path.prefix", "/es") .option("es.port", "443")
From the databricks terminal if I attempt to curl using
I get the expected response, however if I flip it and
It fails with the same error. This also works successfully
so if there is a way to completely omit the port that would be great but it seems like this defaults to 9200 and I haven't figured out how to prevent that.
Any suggestions would be appreciated? Is this a bug that warrants an issue on the GH project?