Hi,
I am trying to insert documents in the index after reading documents from GCP storage. My Elasticsearch flavour on GCP is elastic cloud service hosted by elastic. for this, I am using version 7.5.0.
and my Gradle dependencies look like the below:-
dependencies {
compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.3.4'
compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.3.4'
compile group: 'org.elasticsearch', name: 'elasticsearch-hadoop', version: '7.5.0'
compile group: 'org.elasticsearch',name:'elasticsearch',version:'7.5.0'
}
In code, I am passing username and auth using spark.es.net.http.auth.user & spark.es.net.http.auth.pass property.
So, with the same code and dependency, if I just change the hostname to localhost and port no. to 9200 which is different for elastic cloud service, my code is able to successfully insert the documents in the index in the local elastic cluster. but while trying with elastic cloud hostname and 9243 port,
I am gettting the weird error
Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/sql/api/java/JavaEsSparkSQL
at com.cortex.spark.gcpSparktoES.main(gcpSparktoES.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:890)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.spark.sql.api.java.JavaEsSparkSQL
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
Can you please help me with the solution for it.
For your information , same error is coming when i am submitting the job on dataproc cluster on GCP.