Writing data from Spark to Elasticsearch

Elastic Stack Elasticsearch
1

I am using spark2.0 and using the spark session variable .trying to write some data into elastic search cluster installed in my local machine.code snippet is given below

spark.conf.set("es.index.auto.create", "true")
spark.conf.set("es.nodes", "127.0.0.1")
spark.conf.set("es.port", "9200")
val test = spark.createDataset(Seq(1,2))
EsSpark.saveToEs(test,"abc/def")

Last line is throwing a compilation error .Can some one suggest what is the correct way of calling the saveToEs method?

2

You'll need to transform your Dataset to be compatible with the DataFrame type, which means it must have a schema, and be capable of being transformed into a Dataset of type [Row] to be consumed.

3

thank you James.
I was able to write using SaveToEs method . code is given below
val stats = spark.sql("select ... from table group by col1")
stats.saveToEs("abc/def")

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.