my version is
spark :2.0.2
elasticsearch : 2.3.2
I found es-hadoop-5.0+ support spark 2.0+ with es 5.0
Does it support spark 2.0+ with es2.x.x ?
my version is
spark :2.0.2
elasticsearch : 2.3.2
I found es-hadoop-5.0+ support spark 2.0+ with es 5.0
Does it support spark 2.0+ with es2.x.x ?
ES-Hadoop 5.1 supports Spark 1.3-2.0 and Elasticsearch 1.x to 5.1. If you are having issues with the framework please post a stack trace and a description of the problem you're running into.
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 5, 192.168.1.90): java.lang.NoSuchMethodError: org.apache.spark.TaskContext.addOnCompleteCallback(Lscala/Function0;)V
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:42)
at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:84)
at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:84)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
the error is the above, I have tip in the es-hadoop github.
I guess es 2.3.2 work with spark 2.0.2 is deprecated by es-hadoop because I found elasticsearch-spark-20_2.11-5.1.1.jar still not supports this.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.