How to read/write to Elasticsearch with Apache Spark with scala

How to read/write to Elasticsearch with Apache Spark with scala

Elasticsearch version 5
Spark version 2.2
Any Help ?

What have you tried?

1 Like

i'm trying to write and read to elasticsearch using apache spark
but data written to elasticsearch in format base64

this is the code i'm using to write to elasticsearch

var df = spark.readStream
        .format("kafka")
        .option("kafka.bootstrap.servers", KafkaService.bootstrapServers)
        .option("enable.auto.commit", KafkaService.enableAutoCommit)
        .option("failOnDataLoss", KafkaService.failOnDataLoss)
        .option("startingOffsets", KafkaService.startingOffsets)
        .option("subscribe", topicName)
        .option("group.id", groupId)
        .load()
    
    df.writeStream
    .outputMode(OutputMode.Append) //Only mode for ES
    .format("org.elasticsearch.spark.sql") //es
    .queryName("ElasticSink" + topicName)
    .start(indexName + "/broadcast") //ES index

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.