Structured Streaming - "Failed to find data source: es"

[polomarcus (Paul Leclercq)]

Hi,

I am writing a code example for Structured streaming and elasticsearch-hadoop 6.1.0 and I encounter this error Failed to find data source: es. Would someone know why ?

.format("es")

My complete code example is available here :

Add elasticsearch example

Would "org.elasticsearch.spark.sql" be OK to use ?

[james.baiera]

"org.elasticsearch.spark.sql" should be fine to use as well as "es". Can you share the stack trace that you are seeing? Are you sure that the correct jar is on the classpath?

[polomarcus (Paul Leclercq)]

Thanks for the quick answer.

java.lang.ClassNotFoundException: Failed to find data source: es. Please find packages at http://spark.apache.org/third-party-projects.html
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:549)
	at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)
	at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)
	at org.apache.spark.sql.execution.datasources.DataSource.createSink(DataSource.scala:270)
	at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:286)
	at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:222)
	at elastic.ElasticSink$.writeStream(ElasticSink.scala:19)
	at main.Main$.main(Main.scala:24)
	at main.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: java.lang.ClassNotFoundException: es.DefaultSource
	at java.lang.ClassLoader.findClass(ClassLoader.java:530)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

With this version used : libraryDependencies += "org.elasticsearch" % "elasticsearch-hadoop" % "6.1.0"

Just to be sure, what should be the imported class when we define our query ?

[polomarcus (Paul Leclercq)]

"org.elasticsearch.spark.sql" works fine
But I'm still wondering why I am not able to use "es"