Failed to verify server certificate - APM Go Agent with ECK

Hongbo-Miao · August 8, 2021, 3:34pm

Kibana version:
7.14

Elasticsearch version:
7.14

APM Server version:
7.14

APM Agent language and version:
Go (Gin)
go.elastic.co/apm/module/apmgin v1.13.1

Browser version:
None

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

I am using Elastic Cloud on Kubernetes (ECK) way to install. Here is the step:

I have a folder elastic with these files:

elastic/elastic-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: elastic

elastic/hm-elasticsearch.yaml

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: hm-elasticsearch
  namespace: elastic
spec:
  version: 7.14.0
  nodeSets:
    - name: default
      count: 1
      config:
        node.store.allow_mmap: false

elastic/hm-kibana.yaml

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: hm-kibana
  namespace: elastic
spec:
  version: 7.14.0
  count: 1
  elasticsearchRef:
    name: hm-elasticsearch

elastic/hm-amp.yaml

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: hm-apm
  namespace: elastic
spec:
  version: 7.14.0
  count: 1
  elasticsearchRef:
    name: "hm-elasticsearch"
  kibanaRef:
    name: "hm-kibana"

Then I installed by

kubectl apply --filename=https://download.elastic.co/downloads/eck/1.7.0/crds.yaml
kubectl apply --filename=https://download.elastic.co/downloads/eck/1.7.0/operator.yaml
kubectl apply --filename=elastic  # "elastis" is the folder including files above.

Fresh install or upgraded from other version?
Fresh install

Is there anything special in your setup?
None

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

After install, I can see these services:

➜ kubectl get svc -n elastic
NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
hm-elasticsearch-es-transport   ClusterIP   None           <none>        9300/TCP   4h49m
hm-elasticsearch-es-http        ClusterIP   10.43.222.10   <none>        9200/TCP   4h49m
hm-kibana-kb-http               ClusterIP   10.43.17.34    <none>        5601/TCP   4h49m
hm-apm-apm-http                 ClusterIP   10.43.72.56    <none>        8200/TCP   4h49m
hm-elasticsearch-es-default     ClusterIP   None           <none>        9200/TCP   4h49m

and secrets:

➜ kubectl get secret -n elastic
NAME                                             TYPE                                  DATA   AGE
default-token-hrz7n                              kubernetes.io/service-account-token   3      4h6m
hm-kibana-kibana-user                            Opaque                                1      4h6m
elastic-hm-kibana-kibana-user                    Opaque                                3      4h6m
hm-apm-apm-kb-user                               Opaque                                1      4h6m
hm-apm-apm-user                                  Opaque                                1      4h6m
hm-elasticsearch-es-http-ca-internal             Opaque                                2      4h6m
elastic-hm-apm-apm-kb-user                       Opaque                                3      4h6m
elastic-hm-apm-apm-user                          Opaque                                3      4h6m
hm-kibana-kb-http-ca-internal                    Opaque                                2      4h6m
hm-kibana-kb-http-certs-internal                 Opaque                                3      4h6m
hm-kibana-kb-http-certs-public                   Opaque                                2      4h6m
hm-apm-apm-kibana-ca                             Opaque                                2      4h6m
hm-elasticsearch-es-http-certs-internal          Opaque                                3      4h6m
hm-elasticsearch-es-http-certs-public            Opaque                                2      4h6m
hm-kibana-kb-es-ca                               Opaque                                2      4h6m
hm-apm-apm-http-ca-internal                      Opaque                                2      4h6m
hm-apm-apm-es-ca                                 Opaque                                2      4h6m
hm-elasticsearch-es-transport-ca-internal        Opaque                                2      4h6m
hm-elasticsearch-es-transport-certs-public       Opaque                                1      4h6m
hm-elasticsearch-es-remote-ca                    Opaque                                1      4h6m
hm-elasticsearch-es-elastic-user                 Opaque                                1      4h6m
hm-elasticsearch-es-internal-users               Opaque                                3      4h6m
hm-elasticsearch-es-xpack-file-realm             Opaque                                3      4h6m
hm-apm-apm-http-certs-internal                   Opaque                                3      4h6m
hm-apm-apm-http-certs-public                     Opaque                                2      4h6m
hm-apm-apm-token                                 Opaque                                1      4h6m
hm-apm-apm-config                                Opaque                                1      4h6m
hm-elasticsearch-es-default-es-transport-certs   Opaque                                3      4h6m
hm-elasticsearch-es-default-es-config            Opaque                                1      4h6m
hm-kibana-kb-config                              Opaque                                2      4h6m

Then I try to save the APM public certificate locally by

kubectl get secret hm-elasticsearch-es-http-certs-public --namespace=elastic --output=go-template='{{index .data "tls.crt" | base64decode }}' > data/elastic-apm/tls.crt

I mount this tls.crt to Kubernetes by PersistentVolume, and pass to my Go (Gin) app container by ELASTIC_APM_SERVER_CERT. Here is my full env list I am using for this Go app:

ELASTIC_APM_SERVER_URL: "https://hm-apm-apm-http.elastic:8200"
ELASTIC_APM_ENVIRONMENT: "development"
ELASTIC_APM_LOG_LEVEL: "debug"
ELASTIC_APM_LOG_FILE: "stderr"
ELASTIC_APM_VERIFY_SERVER_CERT: "true"
ELASTIC_APM_SERVER_CERT: "/data/elastic-apm/tls.crt"

I can confirm my certificate be mounted successfully. However, my Elastic APM agent in Go app shows error

{"level":"error","time":"2021-08-08T16:29:22Z","message":"config request failed: sending config request failed: Get "https://hm-apm-apm-http.elastic:8200/config/v1/agents?service.environment=production\u0026service.name=hm-api-server\": failed to verify server certificate"}

{"level":"debug","time":"2021-08-08T16:29:51Z","message":"gathering metrics"}

{"level":"debug","time":"2021-08-08T16:29:52Z","message":"request failed: sending event request failed: Post "https://hm-apm-apm-http.elastic:8200/intake/v2/events\": failed to verify server certificate (next request in ~0s)"}

Am I using right certificate from Kubernetes secrets? Because there are many secrets there.

Any help would be appreciate, thanks!

stephenb · August 8, 2021, 4:52pm

I would think you would use

hm-apm-apm-http-certs-public

Because the agent talks to the APM server not directly to Elasticsearch.

The agent is going to need the apm-token as well that will be in hm-apm-apm-token

Hongbo-Miao · August 8, 2021, 5:06pm

Thank you so much @stephenb , that is a good catch!

Now my error becomes

{"level":"debug","time":"2021-08-08T17:00:09Z","message":"request failed: sending event request failed: Post "https://hm-apm-apm-http.elastic:8200/intake/v2/events\": x509: certificate signed by unknown authority (next request in ~36s)"}

As this is a self signed certificate, I found this answer: X509: certificate signed by unknown authority - #2 by jsoriano

As I am using Elastic Cloud on Kubernetes (ECK) way to install, I am not sure how to change the config.

I also tried to change to

ELASTIC_APM_VERIFY_SERVER_CERT: "false"

for my Go app. However, just like the doc says, if I am using ELASTIC_APM_SERVER_CERT, the ELASTIC_APM_VERIFY_SERVER_CERT will be ignored.

Any guide? Thanks!

Hongbo-Miao · August 8, 2021, 5:22pm

Found this APM agents do not connect · Issue #2657 · elastic/cloud-on-k8s · GitHub
Will do more experiments and report back if I can make it!

stephenb · August 8, 2021, 5:23pm

I am not a Go expert.

I think your choices are:

not validate the ssl cert at all

ELASTIC_APM_VERIFY_SERVER_CERT: "false"

And

Take out ELASTIC_APM_SERVER_CERT

This is probably ok for dev test not sure I would recommend for production.

Create a non self signed certificate.
Add the self signed CA as a trusted CA for you Go app or system it is running on. I do not know how to do that.

Not sure the issue you referenced is the same you issue is a Cert / CA issue

Note : I also fixed your thread topic. You will get more experts to look at your topics if you're more clear in your subject lines.

Oh and BTW @Hongbo-Miao I forget to say... Welcome to the community!

stephenb · August 8, 2021, 6:04pm

@Hongbo-Miao

WOW Think I found something, looks like this was just added and NOT in the Docs Yet!

This is like the other agents where you can specify the CA (Certificate Authority)

This appears to be in Golang Agent 1.13.x +

Looks like the setting is

ELASTIC_APM_SERVER_CA_CERT_FILE

So I think you you get this secret

hm-apm-apm-http-ca-internal

and the put this CA on the client / agent side and provide the fully qualified path to it in the ELASTIC_APM_SERVER_CA_CERT_FILE setting and then that should validate the Cert and CA

You will still need the ELASTIC_APM_SERVER_CERT setting

Give it a try and report back

Its is even mentioned in the 1.13.0 release notes here

Hongbo-Miao · August 8, 2021, 7:26pm

Hi @stephenb thank you so much for the help!

Sorry I missed this sentence early:

The agent is going to need the apm-token as well that will be in hm-apm-apm-token

Here are 3 ways I tested that work:

First to get ELASTIC_APM_SECRET_TOKEN, run

kubectl get secret hm-apm-apm-token --namespace=elastic --output=go-template='{{index .data "secret-token"}}' | base64 -d && echo

Method 1

Change to ELASTIC_APM_VERIFY_SERVER_CERT: "false" and remove ELASTIC_APM_SERVER_CERT, so like

ELASTIC_APM_SERVER_URL: "https://hm-apm-apm-http.elastic:8200"
ELASTIC_APM_ENVIRONMENT: "development"
ELASTIC_APM_LOG_LEVEL: "debug"
ELASTIC_APM_LOG_FILE: "stderr"
ELASTIC_APM_SECRET_TOKEN: "xxx"
ELASTIC_APM_VERIFY_SERVER_CERT: "false"

Method 2

kubectl get secret hm-apm-apm-http-certs-public --namespace=elastic --output=go-template='{{index .data "tls.crt" | base64decode }}' > data/elastic-apm/tls.crt

Mounted tls.crt to the app, and pass these to Go app envs

ELASTIC_APM_SERVER_URL: "https://hm-apm-apm-http.elastic:8200"
ELASTIC_APM_ENVIRONMENT: "development"
ELASTIC_APM_LOG_LEVEL: "debug"
ELASTIC_APM_LOG_FILE: "stderr"
ELASTIC_APM_SECRET_TOKEN: "xxx"
ELASTIC_APM_SERVER_CERT: "/data/elastic-apm/tls.crt"

Method 3

Adding ELASTIC_APM_SERVER_CA_CERT_FILE will still work:

kubectl get secret hm-apm-apm-http-certs-public --namespace=elastic --output=go-template='{{index .data "tls.crt" | base64decode }}' > data/elastic-apm/tls.crt
kubectl get secret hm-apm-apm-http-certs-public --namespace=elastic --output=go-template='{{index .data "ca.crt" | base64decode }}' > data/elastic-apm/ca.crt

Mounted both tls.crt and ca.crt to the app, and pass these to Go app envs

ELASTIC_APM_SERVER_URL: "https://hm-apm-apm-http.elastic:8200"
ELASTIC_APM_ENVIRONMENT: "development"
ELASTIC_APM_LOG_LEVEL: "debug"
ELASTIC_APM_LOG_FILE: "stderr"
ELASTIC_APM_SECRET_TOKEN: "xxx"
ELASTIC_APM_SERVER_CERT: "/data/elastic-apm/tls.crt"
ELASTIC_APM_SERVER_CA_CERT_FILE: "/data/elastic-apm/ca.crt"

stephenb · August 8, 2021, 7:31pm

Cool

Do all 3 methods work ?

I am surprised method 2 does without the CA... Perhaps the cert is full chain including the CA.

But good to know, glad you got it working!

Hongbo-Miao · August 8, 2021, 7:34pm

Yeah, all work.

I had a closer look, for method 2 and method 3 using certificates, if without ELASTIC_APM_SECRET_TOKEN , it won't work.

When I am using certificate, am I supposed to use ELASTIC_APM_SECRET_TOKEN at same time?

If not, then it means my method 2 and 3 still do not work once removing ELASTIC_APM_SECRET_TOKEN. Both with get

{"level":"error","time":"2021-08-08T19:40:40Z","message":"config request failed: request failed with 401 Unauthorized: {"error":"missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'"}"}

stephenb · August 8, 2021, 7:40pm

Yes you always need the token.

That is a separate security check from the SSL certificate verification.

So I guess method 2 works because the tls.cert has the CA included... Full chain

You had this error at one point

"x509: certificate signed by unknown authority"

That usually means you need the CA, but good if you don't, perhaps earlier you pulled the wrong cert.

Hongbo-Miao · August 8, 2021, 7:41pm

Cool, then all three work now! I don't know, I cannot reproduce that

x509: certificate signed by unknown authority

any more even I tried my old config...

Anyway, thanks a lot!

stephenb · August 8, 2021, 7:43pm

Good!

It was probably at bad cert at one point!

So method 2 it is...

This was a good conversation because some people will need the CA and that has been added now.

Good Luck!

axw · August 9, 2021, 1:05am

@stephenb good pickup on the new CA cert config. I've opened docs: document ELASTIC_APM_SERVER_CA_CERT_FILE · Issue #1001 · elastic/apm-agent-go · GitHub to get that documented.

FWIW: method 2 works because it's checking apm-server's certificate matches exactly, rather than verifying the entire chain using the CA cert. i.e. "certificate pinning".

system · August 29, 2021, 9:06pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.