org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: returned [400|Bad Request:]

[m-kiuchi]

Hi, there

When I tried read index from Spark, found error.

py4j.protocol.Py4JJavaError: An error occurred while calling o35.load.
: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [HEAD] on [projects/project] failed; server[104.215.63.49:9200] returned [400|Bad Request:]

Environment:

• Fedora 26
• JDK 1.8.0-141
• Spark 2.2.0
• Elasticsearch 5.5.1

Here is code snipet.

from pyspark.sql import SparkSession
spark = SparkSession \
    .builder \
    .appName("myapp") \
    .config("es.nodes", "111.111.111.111") \
    .getOrCreate()
df = spark.read.format("org.elasticsearch.spark.sql").load("projects/project")

run like this.

$ ~/spark/bin/spark-submit --packages org.elasticsearch:elasticsearch-spark-20_2.11:5.3.1 ./cornw-step1s.py

Any advice is welcome.

[james.baiera]

@m-kiuchi This is fixed in ES-Hadoop v5.5.0. Please upgrade to that version in order to interact with ES 5.5.0+.

[m-kiuchi]

@james.baiera Thanks much. I've succeed to read from elasticsearch !