I am writing a code example for Structured streaming and elasticsearch-hadoop 6.1.0 and I encounter this error Failed to find data source: es. Would someone know why ?
.format("es")
My complete code example is available here :
Would "org.elasticsearch.spark.sql" be OK to use ?
"org.elasticsearch.spark.sql" should be fine to use as well as "es". Can you share the stack trace that you are seeing? Are you sure that the correct jar is on the classpath?
java.lang.ClassNotFoundException: Failed to find data source: es. Please find packages at http://spark.apache.org/third-party-projects.html
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:549)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)
at org.apache.spark.sql.execution.datasources.DataSource.createSink(DataSource.scala:270)
at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:286)
at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:222)
at elastic.ElasticSink$.writeStream(ElasticSink.scala:19)
at main.Main$.main(Main.scala:24)
at main.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: java.lang.ClassNotFoundException: es.DefaultSource
at java.lang.ClassLoader.findClass(ClassLoader.java:530)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
With this version used : libraryDependencies += "org.elasticsearch" % "elasticsearch-hadoop" % "6.1.0"
Just to be sure, what should be the imported class when we define our query ?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.