How to setup X- PACK security feature for ES 7.1.0 installed from ES Helm chart

Hi All,

I am trying to install elasticsearch 7 using helm on kubernetes..

Below is the helm chart of elasticsearch I am using,

However now I am able to setup the elasticsearch and kibana and its running fine. Now I want to use the security feature of x-pack in elasticsearch and kibana and I followed the below link,

I am executing "make" command and below is the error response I am getting,

make

kubectl delete secrets elastic-credentials elastic-certificates elastic-certificate-pem || true &&
vault read -field=value secret/devops-ci/helm-charts/elasticsearch/security/certificates | base64 --decode > elastic-certificates.p12 &&
vault read -field=value secret/devops-ci/helm-charts/elasticsearch/security/certificate-pem | base64 --decode > elastic-certificate.pem &&
kubectl create secret generic elastic-credentials --from-literal=password=changeme --from-literal=username=elastic &&
kubectl create secret generic elastic-certificates --from-file=elastic-certificates.p12 &&
kubectl create secret generic elastic-certificate-pem --from-file=elastic-certificate.pem &&
rm -f elastic-certificates.p12 elastic-certificate.pem
Error from server (NotFound): secrets "elastic-credentials" not found
Error from server (NotFound): secrets "elastic-certificates" not found
Error from server (NotFound): secrets "elastic-certificate-pem" not found
/bin/sh: 2: vault: not found
/bin/sh: 3: vault: not found
secret "elastic-credentials" created
secret "elastic-certificates" created
secret "elastic-certificate-pem" created
helm upgrade --wait --timeout=600 --install --values ./security.yml helm-es-security ../../ ;
Error: UPGRADE FAILED: "helm-es-security" has no deployed releases
Makefile:8: recipe for target 'install' failed
make: *** [install] Error 1

Kindly help me to understand how the security feature setup done in helm charts and correct me if I am doing anything wrong.

Regards,
Ganeshbabu R

Hi!

These examples are also part of the automated testing suite for the charts. It looks like I switched the default makefile target to include pulling the secrets from our vault service.

For now can you just try running make install or helm upgrade --wait --timeout=600 --install --values ./security.yml elasticsearch ../../. That will just do the install without trying to pull in the secrets from elsewhere.

I have made a note to fix up the docs as well to make sure no one else runs into this problem in the future. Thanks a lot for reporting it!

Hi @michael.russell,

Thanks for the information..

I tried executing the below command,

ubuntu@localhost:~/KubernetesCluster/elasticsearch7/helm-charts/elasticsearch/examples/security$ helm upgrade --wait --timeout=600 --install --values ./security.yml elasticsearch ../../.
Release "elasticsearch" does not exist. Installing it now.


Error: release elasticsearch failed: timed out waiting for the condition

Below is the helm status

helm status elasticsearch
LAST DEPLOYED: Thu Jun  6 18:33:28 2019
NAMESPACE: intell-2
STATUS: PENDING_INSTALL

RESOURCES:
==> v1/Service
NAME                      AGE
security-master           9m
security-master-headless  9m

==> v1beta1/StatefulSet
security-master  9m

==> v1beta1/PodDisruptionBudget
security-master-pdb  9m

==> v1/Pod(related)

NAME               READY  STATUS            RESTARTS  AGE
security-master-0  1/2    CrashLoopBackOff  6         9m
security-master-1  1/2    CrashLoopBackOff  6         9m
security-master-2  1/2    CrashLoopBackOff  6         9m

==> v1/ConfigMap

NAME                    AGE
security-master-config  9m

==> v1/StorageClass
elasticsearch7  9m


NOTES:
1. Watch all cluster members come up.
  $ kubectl get pods --namespace=intell-2 -l app=security-master -w
2. Test cluster health using Helm test.
  $ helm test elasticsearch

ubuntu@ip-localhost:~$ kubectl get pods --namespace=intell-2 -l app=security-master -w
NAME                READY     STATUS             RESTARTS   AGE
security-master-0   1/2       CrashLoopBackOff   6          10m
security-master-1   1/2       CrashLoopBackOff   6          10m
security-master-2   1/2       CrashLoopBackOff   6          10m

Below is the pod describe response,

kubectl describe pod security-master-0 -n intell-2

Events:
  Type     Reason                  Age                From                                    Message
  ----     ------                  ----               ----                                    -------
  Normal   Scheduled               12m                default-scheduler                       Successfully assigned intell-2/security-master-0 to ip-localhost.ec2.internal
  Normal   SuccessfulAttachVolume  12m                attachdetach-controller                 AttachVolume.Attach succeeded for volume "pvc-894a4-8811-11e9-82aa-0a8698864024"
  Normal   Pulled                  11m                kubelet, ip-localhost.ec2.internal  Container image "docker.elastic.co/elasticsearch/elasticsearch:7.1.0" already present on machine
  Normal   Created                 11m                kubelet, ip-localhost.ec2.internal  Created container
  Normal   Started                 11m                kubelet, ip-localhost.ec2.internal  Started container
  Normal   Pulled                  11m                kubelet, ip-localhost.ec2.internal  Container image "docker.elastic.co/elasticsearch/elasticsearch:7.1.0" already present on machine
  Normal   Created                 11m                kubelet, ip-localhost.ec2.internal  Created container
  Normal   Started                 11m                kubelet, ip-localhost.ec2.internal  Started container
  Normal   Created                 10m (x4 over 11m)  kubelet, ip-localhost.ec2.internal  Created container
  Normal   Started                 10m (x4 over 11m)  kubelet, ip-localhost.ec2.internal  Started container
  Normal   Pulled                  9m (x5 over 11m)   kubelet, ip-localhost.ec2.internal  Container image "docker.elastic.co/elasticsearch/elasticsearch:7.1.0" already present on machine
  Warning  BackOff                 1m (x38 over 11m)  kubelet, ip-localhost.ec2.internal  Back-off restarting failed container

Checked the logs of security-master-0 pod and found the below errors,

{"type": "server", "timestamp": "2019-06-06T18:51:49,628+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "security", "node.name": "security-master-0", "message": "uncaught exception in thread [main]" ,
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.0.jar:7.1.0]",

"Caused by: java.lang.reflect.InvocationTargetException",
"at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]",

"Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory",
"at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]",
"at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]",

"Caused by: java.io.IOException: Short read of DER length",
"at sun.security.util.DerInputStream.getLength(DerInputStream.java:588) ~[?:?]",
"at sun.security.util.DerValue.init(DerValue.java:390) ~[?:?]",
"at sun.security.util.DerValue.(DerValue.java:331) ~[?:?]",
"at sun.security.util.DerValue.(DerValue.java:344) ~[?:?]",

Please correct me if I am doing anything wrong and let me know your thoughts.

Regards,
Ganeshbabu R

This makes it sound like the certificates wasn't generated properly or isn't in the expected format. Did you get any errors when running through the steps to generate the certificates?

You could also try checking that the certificates are valid with:

kubectl get secret elastic-certificates -o jsonpath='{.data.elastic-certificates\.p12}' | base64 --decode | openssl pkcs12 -info

Hey!!! This helm/charts is extemely miserble that i ever used. Any deployment can't work correctly because, the pkcs12 file doesn't work with the other pods on elastic cluster

SSL connection to https://127.0.0.1:9200/_security/_authenticate?pretty failed: No subject alternative names present
Please check the elasticsearch SSL settings under xpack.security.http.ssl.
ERROR: Failed to establish SSL connection to elasticsearch at https://127.0.0.1:9200/_security/_authenticate?pretty.

This shown very mistake with this helm/charts this elasticsearch can't working for any reason. Now on this time no any solution to fix this problem because, when created new CA and certificate with SANs (Subject Alternative Names) and apply on its. Not working the pod is hang and can't do anything. If delete any pod on this statefulset the IP Address was change and certificate can't used for new pods.

Elastic Cloud on K8S can working completely but it's an alpha version . It had many limitation for using in this time (can't customize with dedicated node).

Hi @michael.russell

I have a doubt in generating the certificates, but the official docs saying use the elasticsearch-certutil ca command to generate ca & certificate file

bin/elasticsearch-certutil ca
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

How should i generate the certificates (elastic-certificates.p12) before I executing the kubernetes secrets for authentication,

Create Kubernetes secrets for authentication credentials and certificates

kubectl create secret generic elastic-credentials --from-literal=password=changeme --from-literal=username=elastic
kubectl create secret generic elastic-certificates --from-file=elastic-certificates.p12

Do I need to use openssl for generating node certificates?

Kindly share your thoughts..

Regards,
Ganeshbabu R

Hi @michael.russell,

I tried generating certificates using the openssl command,

below is the steps I tried,

ubuntu@ip-172-16-27-109:~/KubernetesCluster/elasticsearch7/helm-charts$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /home/ubuntu/KubernetesCluster/elasticsearch7/helm-charts/elastic.key -out /home/ubuntu/KubernetesCluster/elasticsearch7/helm-charts/elastic.crt
Generating a 2048 bit RSA private key
.....................................+++
...........+++
writing new private key to '/home/ubuntu/KubernetesCluster/elasticsearch7/helm-charts/elastic.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:Bangalore
Locality Name (eg, city) []:Karnataka
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Microland
Organizational Unit Name (eg, section) []:Automation
Common Name (e.g. server FQDN or YOUR name) []:security-master
Email Address []:abc@microland.com

I gave the Common Name as "security-master" because the name of elasticsearch service as same then I tried to converting to .p12 format by using the below steps,

  1. I copied both key & crt files into the txt file and execute the below command
    openssl pkcs12 -export -in elastic.txt -out elastic-certificates.p12

Now I have created Kubernetes secrets for authentication credentials and certificates and then executed the below helm command to setup elasticsearch with security enabled,

helm upgrade --namespace intelligeni-2 --wait --timeout=600 --install --values /home/ubuntu/KubernetesCluster/elasticsearch7/helm-charts/elasticsearch/examples/security/security.yml elasticsearch /home/ubuntu/KubernetesCluster/elasticsearch7/helm-charts/elasticsearch

Below is the helm status elasticsearch

ubuntu@ip-172-16-27-109:~$ helm status elasticsearch
LAST DEPLOYED: Wed Jun 12 09:57:59 2019
NAMESPACE: intelligeni-2
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/PodDisruptionBudget
NAME                 AGE
security-master-pdb  1d

==> v1/Pod(related)

NAME               READY  STATUS   RESTARTS  AGE
security-master-0  2/2    Running  0         1d
security-master-1  2/2    Running  0         1d
security-master-2  2/2    Running  0         1d

==> v1/ConfigMap

NAME                    AGE
security-master-config  1d

==> v1/StorageClass
elasticsearch7  1d

==> v1/Service
security-master           1d
security-master-headless  1d

==> v1beta1/StatefulSet
security-master  1d


NOTES:
1. Watch all cluster members come up.
  $ kubectl get pods --namespace=intelligeni-2 -l app=security-master -w
2. Test cluster health using Helm test.
  $ helm test elasticsearch

I went inside one of the elasticsearch pod kubectl exec -it security-master-0 bash -n and to check the cluster health with the authentication,

[root@security-master-0 elasticsearch]# curl -u elastic:changeme http://security-master:9200/_cluster/health?pretty=true
curl: (52) Empty reply from server
[root@security-master-0 elasticsearch]# curl -u elastic:changeme http://localhost:9200
curl: (52) Empty reply from server

Can you please tell me why the response is showing as empty reply from server?

[root@security-master-0 elasticsearch]# curl -u elastic:changeme https://security-master:9200/_cluster/health?pretty=true
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

If I change the protocol to https I am getting the above response but now sure why certificate verification is failed?

[root@security-master-0 elasticsearch]# curl -k -u elastic:changeme https://security-master:9200/_cluster/health?pretty=true
{
  "cluster_name" : "security",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 3,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

But if i add -k or --insecure I can able to verify the cluster health with the authentication but still I think certificate verification is not doing properly. Is it because of self signed certificate creation?

Please correct me if I am doing anything wrong and let me know your thoughts.

Thanks,
Ganeshbabu R

But if i add -k or --insecure I can able to verify the cluster health with the authentication but still I think certificate verification is not doing properly. Is it because of self signed certificate creation?

Nope, it looks like this is working perfectly! You are getting a warning about it being a self signed certificate because it is a self signed certificate :). To get curl to work without ignoring certificate warnings you will need to run curl --cacert /path/to/your/self/signed/ca.whatever.