Spark on k8s can't authenticate to ECK

Hello everyone,

I am very excited by using Elasticsearch on Kubernetes with Spark but I cant' authenticate my job to it.
I successfully installed ECK (without TLS => for test) on my GKE cluster with the following template (thanks to Helm):

 apiVersion: v1
 kind: Secret
   name: smart-agriculture-elasticsearch-es-elastic-user # I override the default password for user "elastic" created by ECK during its installation
   namespace: dev
 type: Opaque
   elastic: dG90bwo= # password is "toto" in base64
 kind: Elasticsearch
   name: smart-agriculture-elasticsearch
   namespace: dev
   version: 7.6.1
     - name: default
       count: 1
         node.master: true true
         node.ingest: true false
         disabled: true

Then I want to use my spark job to connect to my Elasticsearch cluster, here is my elasticsearch dependency and my hello world code in scala




import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.elasticsearch.spark._

   object ElasticSparkHelloWorld {
     def main(args: Array[String]) {

   val conf = new SparkConf().setAppName("spark-es-to-parquet").setMaster("k8s://")
   conf.set("kubernetes.namespace", "dev")
   conf.set("kubernetes.authenticate.driver.serviceAccountName", "spark-sa") # searvice account created in another template, it works !
   conf.set("", "false")
   conf.set("es.nodes.wan.only", "true")
   conf.set("es.nodes", "http://smart-agriculture-elasticsearch-es-http")
   conf.set("es.port", "9200")
   conf.set("", "elastic") # user
   conf.set("", "toto") # password

   val sc = new SparkContext(conf)

   val numbers = Map("one" -> 1, "two" -> 2, "three" -> 3)
   val airports = Map("arrival" -> "Otopeni", "SFO" -> "San Fran")

     Seq(numbers, airports)


However, I get the following error in spark that I don't really understand:

Caused by: security_exception: unable to authenticate user [elastic] for REST request [/]

In Elasticsearch logs, I have:

Authentication to realm file1 failed - Password authentication failed for elastic

Does anyone know how to solve this or have information (links, doc..) that I can use ?

Thanks !

Overriding the built-in elastic users password in the secret is not an officially supported feature at this time. Also keep in mind that the elastic user is a superuser and should probably be reserved for admin use cases or similar.

Consider instead:

  • Using the native realm and creating a dedicated user there with a password you control
  • If you really want to use the generated elastic user, why not mount the secret we provide into your application pod as an environment variable or similar?
  • Finally we are working on exposing the so called file realm in a future release of ECK

Thanks @pebrc for your reply.

How can I manipulate native realm with ECK, is it possible or you are still working on it for a future release ? I am confused sorry !
Actually, I used at the beginning secret to provide password to my Pod but I got the same error.

I stoped to override the password for user "elastic" and now it works ! I am looking forward to this futur release.