Prefix and port appear flipped in es-hadoop implementation

Elastic Stack Elasticsearch
1

Attempting to read/write with es-hadoop however I am getting the following error in Databricks

EsHadoopInvalidRequest: [HEAD] on [index_name] failed; server [https://serveraddress.com/es:443] returned [405|Method Not Allowed:]

Note the port comes after the prefix which contradicts what the documentation says here for prefix

es.nodes.path.prefix (default empty)
Prefix to add to all requests made to Elasticsearch. Useful in environments where the cluster is proxied/routed under a certain path. For example, if the cluster is located at someaddress:someport/custom/path/prefix, one would set es.nodes.path.prefix to /custom/path/prefix

My config in databricks just looks like

spark.read.format("org.elasticsearch.spark.sql")
    .option("es.nodes", es_url)
    .option("es.nodes.wan.only","true")
    .option("es.nodes.path.prefix", "/es")
    .option("es.port", "443")

From the databricks terminal if I attempt to curl using

curl https://serveraddress.com:443/es/_search

I get the expected response, however if I flip it and

curl https://serveraddress.com/es:443/_search

It fails with the same error. This also works successfully

curl https://serveraddress.com/es/_search

so if there is a way to completely omit the port that would be great but it seems like this defaults to 9200 and I haven't figured out how to prevent that.

Any suggestions would be appreciated? Is this a bug that warrants an issue on the GH project?

2

This is weird and it looks like a bug, maybe it is better to open an Github issue with instructions to reproduce this bug.

But in the mean time you can ignore the es.port and pass the port directly in the node url, something like serveraddress.com:443.

3

Awesome, that works regarding passing the port in the URL. I'm kicking myself for not thinking to have attempted that. Will open a github issue with steps to reproduce

4

After a brief look at the code, I'm not sure how this happens -- it appears to correctly add the prefix after the port. Steps to reproduce in a ticket will be great. Thanks.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.