Spark SQL types are not handled through basic RDD saveToEs() calls

Elastic Stack Elasticsearch
1

I am trying to a simple operation wherein I am running a Spark job to persist a DataFrame to ElasticSearch.
ElasticSearch Version : 5.3.0
Scala Version - 2.12
Spark Version - 3.0.0
On Trying this code, I get an error

java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;

'Person person1 = new Person();
person1.setName("Piyali");
person1.setColour("Blue");

        Person person2 = new Person();
        person2.setName("Shouvik");
        person2.setColour("Black");

        List<Person> personList = new ArrayList<Person>();
        personList.add(person1);
        personList.add(person2);

        Dataset<Row> dataFrame = sparkSession.createDataFrame(personList, Person.class);

    
        JavaEsSparkSQL.saveToEs(dataFrame, "personList/doc");'

Whereas on trying the below code I get this error
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Spark SQL types are not handled through basic RDD saveToEs() calls; typically this is a mistake(as the SQL schema will be ignored). Use 'org.elasticsearch.spark.sql' package instead
at org.elasticsearch.spark.serialization.ScalaValueWriter.org$elasticsearch$spark$serialization$ScalaValueWriter$$doWrite(ScalaValueWriter.scala:111) ~[elasticsearch-hadoop-5.3.0.jar:5.3.0]
at org.elasticsearch.spark.serialization.ScalaValueWriter.write(ScalaValueWriter.scala:37) ~[elasticsearch-hadoop-5.3.0.jar:5.3.0]

'Person person1 = new Person();
person1.setName("Piyali");
person1.setColour("Blue");

        Person person2 = new Person();
        person2.setName("Shouvik");
        person2.setColour("Black");

        List<Person> personList = new ArrayList<Person>();
        personList.add(person1);
        personList.add(person2);

        Dataset<Row> dataFrame = sparkSession.createDataFrame(personList, Person.class);

      
        JavaEsSpark.saveToEs(dataFrame.toJavaRDD(), "personList/doc");'

Can somebody please help understand where am I going wrong. I am very new to both Spark and ElasticSearch . Any help will be highly appreciated.

Thanks In Advance.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.