Structured Streaming - "Failed to find data source: es"

Hi,

I am writing a code example for Structured streaming and elasticsearch-hadoop 6.1.0 and I encounter this error Failed to find data source: es. Would someone know why ?

.format("es")

My complete code example is available here :

Would "org.elasticsearch.spark.sql" be OK to use ?

1 Like

"org.elasticsearch.spark.sql" should be fine to use as well as "es". Can you share the stack trace that you are seeing? Are you sure that the correct jar is on the classpath?

Thanks for the quick answer.

java.lang.ClassNotFoundException: Failed to find data source: es. Please find packages at http://spark.apache.org/third-party-projects.html
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:549)
	at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)
	at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)
	at org.apache.spark.sql.execution.datasources.DataSource.createSink(DataSource.scala:270)
	at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:286)
	at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:222)
	at elastic.ElasticSink$.writeStream(ElasticSink.scala:19)
	at main.Main$.main(Main.scala:24)
	at main.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: java.lang.ClassNotFoundException: es.DefaultSource
	at java.lang.ClassLoader.findClass(ClassLoader.java:530)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

With this version used : libraryDependencies += "org.elasticsearch" % "elasticsearch-hadoop" % "6.1.0"

Just to be sure, what should be the imported class when we define our query ?

"org.elasticsearch.spark.sql" works fine :face_with_monocle:
But I'm still wondering why I am not able to use "es"

2 Likes

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.