Structured Streaming - "Failed to find data source: es"


(Paul Leclercq) #1

Hi,

I am writing a code example for Structured streaming and elasticsearch-hadoop 6.1.0 and I encounter this error Failed to find data source: es. Would someone know why ?

.format("es")

My complete code example is available here :

Would "org.elasticsearch.spark.sql" be OK to use ?


(James Baiera) #2

"org.elasticsearch.spark.sql" should be fine to use as well as "es". Can you share the stack trace that you are seeing? Are you sure that the correct jar is on the classpath?


(Paul Leclercq) #3

Thanks for the quick answer.

java.lang.ClassNotFoundException: Failed to find data source: es. Please find packages at http://spark.apache.org/third-party-projects.html
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:549)
	at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)
	at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)
	at org.apache.spark.sql.execution.datasources.DataSource.createSink(DataSource.scala:270)
	at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:286)
	at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:222)
	at elastic.ElasticSink$.writeStream(ElasticSink.scala:19)
	at main.Main$.main(Main.scala:24)
	at main.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: java.lang.ClassNotFoundException: es.DefaultSource
	at java.lang.ClassLoader.findClass(ClassLoader.java:530)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

With this version used : libraryDependencies += "org.elasticsearch" % "elasticsearch-hadoop" % "6.1.0"

Just to be sure, what should be the imported class when we define our query ?


Structured Streaming ES Sink
(Paul Leclercq) #4

"org.elasticsearch.spark.sql" works fine :face_with_monocle:
But I'm still wondering why I am not able to use "es"


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.