# Spark on k8s can't authenticate to ECK
**URL:** https://discuss.elastic.co/t/spark-on-k8s-cant-authenticate-to-eck/222764
**Category:** Elastic Cloud on Kubernetes (ECK)
**Created:** [March 9, 2020, 4:26pm UTC](https://discuss.elastic.co/t/spark-on-k8s-cant-authenticate-to-eck/222764 "2020-03-09T16:26:29Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@YaSe](https://discuss.elastic.co/u/YaSe)
#### Post date: [March 9, 2020, 4:26pm UTC](https://discuss.elastic.co/t/spark-on-k8s-cant-authenticate-to-eck/222764/1 "2020-03-09T16:26:29Z")
Hello everyone,
I am very excited by using Elasticsearch on Kubernetes with Spark but I cant' authenticate my job to it.
I successfully installed ECK (without TLS =\> for test) on my GKE cluster with the following template (thanks to Helm):
```
apiVersion: v1
kind: Secret
metadata:
name: smart-agriculture-elasticsearch-es-elastic-user # I override the default password for user "elastic" created by ECK during its installation
namespace: dev
type: Opaque
data:
elastic: dG90bwo= # password is "toto" in base64
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: smart-agriculture-elasticsearch
namespace: dev
spec:
version: 7.6.1
nodeSets:
- name: default
count: 1
config:
node.master: true
node.data: true
node.ingest: true
node.store.allow_mmap: false
http:
tls:
selfSignedCertificate:
disabled: true
```
Then I want to use my spark job to connect to my Elasticsearch cluster, here is my elasticsearch dependency and my hello world code in scala
--
```
org.elasticsearch
elasticsearch-spark-20_2.11
7.6.1
```
--
```
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.elasticsearch.spark._
object ElasticSparkHelloWorld {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("spark-es-to-parquet").setMaster("k8s://https://10.0.0.1:443")
conf.set("executor.instances","2")
conf.set("kubernetes.namespace", "dev")
conf.set("kubernetes.authenticate.driver.serviceAccountName", "spark-sa") # searvice account created in another template, it works !
conf.set("es.index.auto.create", "false")
conf.set("es.nodes.wan.only", "true")
conf.set("es.nodes", "http://smart-agriculture-elasticsearch-es-http")
conf.set("es.port", "9200")
conf.set("es.net.http.auth.user", "elastic") # user
conf.set("es.net.http.auth.pass", "toto") # password
val sc = new SparkContext(conf)
val numbers = Map("one" -> 1, "two" -> 2, "three" -> 3)
val airports = Map("arrival" -> "Otopeni", "SFO" -> "San Fran")
sc.makeRDD(
Seq(numbers, airports)
).saveToEs("spark/docs")
}
```
}
However, I get the following error in spark that I don't really understand:
```
Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: security_exception: unable to authenticate user [elastic] for REST request [/]
```
In Elasticsearch logs, I have:
```
Authentication to realm file1 failed - Password authentication failed for elastic
```
Does anyone know how to solve this or have information (links, doc..) that I can use ?
Thanks !
---
### Author:  [@pebrc](https://discuss.elastic.co/u/pebrc)
#### Post date: [March 10, 2020, 9:12am UTC](https://discuss.elastic.co/t/spark-on-k8s-cant-authenticate-to-eck/222764/2 "2020-03-10T09:12:03Z")
> [@YaSe](#):
>
> ```auto
> apiVersion: v1
> kind: Secret
> metadata:
> name: smart-agriculture-elasticsearch-es-elastic-user # I override the default password for user "elastic" created by ECK during its installation
> namespace: dev
> type: Opaque
> data:
> elastic: dG90bwo= # password is "toto" in base64
>
> ```
Overriding the built-in elastic users password in the secret is not an officially supported feature at this time. Also keep in mind that the `elastic` user is a superuser and should probably be reserved for admin use cases or similar.
Consider instead:
- Using the [native realm](https://www.elastic.co/guide/en/elasticsearch/reference/current/native-realm.html#native-realm-configuration) and creating a dedicated user there with a password you control
- If you really want to use the generated `elastic` user, why not mount the secret we provide into your application pod as an environment variable or similar?
- Finally we are working on exposing the so called file realm in a [future release of ECK](https://github.com/elastic/cloud-on-k8s/pull/2682)
---
### Author:  [@YaSe](https://discuss.elastic.co/u/YaSe)
#### Post date: [March 10, 2020, 11:10am UTC](https://discuss.elastic.co/t/spark-on-k8s-cant-authenticate-to-eck/222764/3 "2020-03-10T11:10:56Z")
Thanks @pebrc for your reply.
How can I manipulate native realm with ECK, is it possible or you are still working on it for a future release ? I am confused sorry !
Actually, I used at the beginning secret to provide password to my Pod but I got the same error.
---
### Author:  [@YaSe](https://discuss.elastic.co/u/YaSe)
#### Post date: [March 10, 2020, 1:18pm UTC](https://discuss.elastic.co/t/spark-on-k8s-cant-authenticate-to-eck/222764/4 "2020-03-10T13:18:06Z")
I stoped to override the password for user "elastic" and now it works ! I am looking forward to [this futur release](https://github.com/elastic/cloud-on-k8s/pull/2682).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 4, 2022, 7:48am UTC](https://discuss.elastic.co/t/spark-on-k8s-cant-authenticate-to-eck/222764/5 "2022-11-04T07:48:30Z")