EsHadoopIllegalArgumentException: Detected type name in resource On elastic 8.8.2

Elastic Stack Elasticsearch
1

Hi
I upgraded my elastic server fom 7.9.3 to 8.8.2

With my previous configuration :

    implementation("org.elasticsearch:elasticsearch-spark-20_2.11:7.9.3") {
        exclude("org.apache.spark")
    }

This code was working (Dynamic/multi resource):

    mySparkDataframe.saveToEs("indexname/{year}")

year is a field in my dataframe

But when i upgraded the elastic server to 8.8.2
my new conf :

    implementation("org.elasticsearch:elasticsearch-spark-20_2.11:8.2.2") {
        exclude("org.apache.spark")
    }

But the same code throws this exception :

User class threw exception: java.lang.Exception: Error(s) during treatment of stream data org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 (TID 1109, host executor 1): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Detected type name in resource [mySparkDataframe/{year}]. Remove type name to continue.

at org.elasticsearch.hadoop.rest.Resource.(Resource.java:88)

at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:595)

at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:71)

at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:108)

at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:108)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

at org.apache.spark.scheduler.Task.run(Task.scala:121)

at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)

at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Can anyone help ?

thanks

2

Hi @Joachim_Rodrigues. It looks like you were using Elasticsearch types (maybe not intentionally?). Elastic began planning to remove them back in 5.0. As of 7.0 I think that support for them was pretty much gone, but most APIs didn't blow up if they were present. As of 8.0 they are no longer supported at all.
You can read a little more about the removal of support for types over time in:

If you're just trying to make sure that data for different years goes into a different place, you might just consider changing your code to something like mySparkDataframe.saveToEs("indexname-{year}"). That would give you one index per year (rather than one type per year in a single index). Does that work? Or is there something else that you were getting from types?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.