# Failed to verify server certificate - APM Go Agent with ECK

**URL:** <https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738>\
**Category:** APM\
**Tags:** elastic-stack-security, go\
**Created:** [August 8, 2021, 3:34pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738 "2021-08-08T15:34:39Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hongbo-Miao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hongbo-miao/32/92664_2.png) [@Hongbo-Miao](https://discuss.elastic.co/u/Hongbo-Miao)\
**Post date:** [August 8, 2021, 3:34pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/1 "2021-08-08T15:34:39Z")

</div>

**Kibana version** :  
7.14

**Elasticsearch version** :  
7.14

**APM Server version** :  
7.14

**APM Agent language and version** :  
Go (Gin)  
[go.elastic.co/apm/module/apmgin](http://go.elastic.co/apm/module/apmgin) v1.13.1

**Browser version** :  
None

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**:

I am using Elastic Cloud on Kubernetes (ECK) way to install. Here is the step:

I have a folder elastic with these files:

_elastic/elastic-namespace.yaml_

```auto
apiVersion: v1
kind: Namespace
metadata:
  name: elastic

```

_elastic/hm-elasticsearch.yaml_

```auto
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: hm-elasticsearch
  namespace: elastic
spec:
  version: 7.14.0
  nodeSets:
    - name: default
      count: 1
      config:
        node.store.allow_mmap: false

```

_elastic/hm-kibana.yaml_

```auto
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: hm-kibana
  namespace: elastic
spec:
  version: 7.14.0
  count: 1
  elasticsearchRef:
    name: hm-elasticsearch

```

_elastic/hm-amp.yaml_

```auto
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: hm-apm
  namespace: elastic
spec:
  version: 7.14.0
  count: 1
  elasticsearchRef:
    name: "hm-elasticsearch"
  kibanaRef:
    name: "hm-kibana"

```

Then I installed by

```auto
kubectl apply --filename=https://download.elastic.co/downloads/eck/1.7.0/crds.yaml
kubectl apply --filename=https://download.elastic.co/downloads/eck/1.7.0/operator.yaml
kubectl apply --filename=elastic # "elastis" is the folder including files above.

```

**Fresh install or upgraded from other version?**  
Fresh install

**Is there anything special in your setup?**  
None

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

After install, I can see these services:

```auto
➜ kubectl get svc -n elastic
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hm-elasticsearch-es-transport ClusterIP None <none> 9300/TCP 4h49m
hm-elasticsearch-es-http ClusterIP 10.43.222.10 <none> 9200/TCP 4h49m
hm-kibana-kb-http ClusterIP 10.43.17.34 <none> 5601/TCP 4h49m
hm-apm-apm-http ClusterIP 10.43.72.56 <none> 8200/TCP 4h49m
hm-elasticsearch-es-default ClusterIP None <none> 9200/TCP 4h49m

```

and secrets:

```auto
➜ kubectl get secret -n elastic
NAME TYPE DATA AGE
default-token-hrz7n kubernetes.io/service-account-token 3 4h6m
hm-kibana-kibana-user Opaque 1 4h6m
elastic-hm-kibana-kibana-user Opaque 3 4h6m
hm-apm-apm-kb-user Opaque 1 4h6m
hm-apm-apm-user Opaque 1 4h6m
hm-elasticsearch-es-http-ca-internal Opaque 2 4h6m
elastic-hm-apm-apm-kb-user Opaque 3 4h6m
elastic-hm-apm-apm-user Opaque 3 4h6m
hm-kibana-kb-http-ca-internal Opaque 2 4h6m
hm-kibana-kb-http-certs-internal Opaque 3 4h6m
hm-kibana-kb-http-certs-public Opaque 2 4h6m
hm-apm-apm-kibana-ca Opaque 2 4h6m
hm-elasticsearch-es-http-certs-internal Opaque 3 4h6m
hm-elasticsearch-es-http-certs-public Opaque 2 4h6m
hm-kibana-kb-es-ca Opaque 2 4h6m
hm-apm-apm-http-ca-internal Opaque 2 4h6m
hm-apm-apm-es-ca Opaque 2 4h6m
hm-elasticsearch-es-transport-ca-internal Opaque 2 4h6m
hm-elasticsearch-es-transport-certs-public Opaque 1 4h6m
hm-elasticsearch-es-remote-ca Opaque 1 4h6m
hm-elasticsearch-es-elastic-user Opaque 1 4h6m
hm-elasticsearch-es-internal-users Opaque 3 4h6m
hm-elasticsearch-es-xpack-file-realm Opaque 3 4h6m
hm-apm-apm-http-certs-internal Opaque 3 4h6m
hm-apm-apm-http-certs-public Opaque 2 4h6m
hm-apm-apm-token Opaque 1 4h6m
hm-apm-apm-config Opaque 1 4h6m
hm-elasticsearch-es-default-es-transport-certs Opaque 3 4h6m
hm-elasticsearch-es-default-es-config Opaque 1 4h6m
hm-kibana-kb-config Opaque 2 4h6m

```

Then I try to save the APM public certificate locally by

```auto
kubectl get secret hm-elasticsearch-es-http-certs-public --namespace=elastic --output=go-template='{{index .data "tls.crt" | base64decode }}' > data/elastic-apm/tls.crt

```

I mount this **tls.crt** to Kubernetes by `PersistentVolume`, and pass to my Go (Gin) app container by `ELASTIC_APM_SERVER_CERT`. Here is my full env list I am using for this Go app:

```auto
ELASTIC_APM_SERVER_URL: "https://hm-apm-apm-http.elastic:8200"
ELASTIC_APM_ENVIRONMENT: "development"
ELASTIC_APM_LOG_LEVEL: "debug"
ELASTIC_APM_LOG_FILE: "stderr"
ELASTIC_APM_VERIFY_SERVER_CERT: "true"
ELASTIC_APM_SERVER_CERT: "/data/elastic-apm/tls.crt"

```

I can confirm my certificate be mounted successfully. However, my Elastic APM agent in Go app shows error

> {"level":"error","time":"2021-08-08T16:29:22Z","message":"config request failed: sending config request failed: Get "[https://hm-apm-apm-http.elastic:8200/config/v1/agents?service.environment=production\u0026service.name=hm-api-server\](https://hm-apm-apm-http.elastic:8200/config/v1/agents?service.environment=production%5Cu0026service.name=hm-api-server%5C)": failed to verify server certificate"}
> 
> {"level":"debug","time":"2021-08-08T16:29:51Z","message":"gathering metrics"}
> 
> {"level":"debug","time":"2021-08-08T16:29:52Z","message":"request failed: sending event request failed: Post "[https://hm-apm-apm-http.elastic:8200/intake/v2/events\](https://hm-apm-apm-http.elastic:8200/intake/v2/events%5C)": failed to verify server certificate (next request in ~0s)"}

Am I using right certificate from Kubernetes secrets? Because there are many secrets there.

Any help would be appreciate, thanks!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 8, 2021, 4:52pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/2 "2021-08-08T16:52:11Z")

</div>

I would think you would use

`hm-apm-apm-http-certs-public`

Because the agent talks to the APM server not directly to Elasticsearch.

The agent is going to need the apm-token as well that will be in `hm-apm-apm-token`

---

<div class="post-metadata">

**Author:** ![Hongbo-Miao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hongbo-miao/32/92664_2.png) [@Hongbo-Miao](https://discuss.elastic.co/u/Hongbo-Miao)\
**Post date:** [August 8, 2021, 5:06pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/3 "2021-08-08T17:06:24Z")

</div>

Thank you so much @stephenb , that is a good catch!

Now my error becomes

> {"level":"debug","time":"2021-08-08T17:00:09Z","message":"request failed: sending event request failed: Post "[https://hm-apm-apm-http.elastic:8200/intake/v2/events\](https://hm-apm-apm-http.elastic:8200/intake/v2/events%5C)": x509: certificate signed by unknown authority (next request in ~36s)"}

As this is a self signed certificate, I found this answer: [X509: certificate signed by unknown authority - #2 by jsoriano](https://discuss.elastic.co/t/x509-certificate-signed-by-unknown-authority/156815/2)

As I am using Elastic Cloud on Kubernetes (ECK) way to install, I am not sure how to change the config.

I also tried to change to

```auto
ELASTIC_APM_VERIFY_SERVER_CERT: "false"

```

for my Go app. However, just like [the doc](https://www.elastic.co/guide/en/apm/agent/go/current/configuration.html#config-verify-server-cert) says, if I am using `ELASTIC_APM_SERVER_CERT`, the `ELASTIC_APM_VERIFY_SERVER_CERT` will be ignored.

Any guide? Thanks!

---

<div class="post-metadata">

**Author:** ![Hongbo-Miao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hongbo-miao/32/92664_2.png) [@Hongbo-Miao](https://discuss.elastic.co/u/Hongbo-Miao)\
**Post date:** [August 8, 2021, 5:22pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/4 "2021-08-08T17:22:00Z")

</div>

Found this [APM agents do not connect · Issue #2657 · elastic/cloud-on-k8s · GitHub](https://github.com/elastic/cloud-on-k8s/issues/2657)  
Will do more experiments and report back if I can make it!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 8, 2021, 5:23pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/5 "2021-08-08T17:23:45Z")

</div>

I am not a Go expert.

I think your choices are:

1. not validate the ssl cert at all

`ELASTIC_APM_VERIFY_SERVER_CERT: "false"`

And

Take out `ELASTIC_APM_SERVER_CERT`

This is probably ok for dev test not sure I would recommend for production.

1. Create a non self signed certificate.

2. Add the self signed CA as a trusted CA for you Go app or system it is running on. I do not know how to do that.

Not sure the issue you referenced is the same you issue is a Cert / CA issue

Note : I also fixed your thread topic. You will get more experts to look at your topics if you're more clear in your subject lines.

Oh and BTW @Hongbo-Miao I forget to say... Welcome to the community!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 8, 2021, 6:04pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/6 "2021-08-08T18:04:21Z")

</div>

@Hongbo-Miao

WOW Think I found something, looks like this was just added and NOT in the Docs Yet!

This is like the other agents where you can specify the CA (Certificate Authority)

> <https://github.com/elastic/apm-agent-go/commit/365b51f7d171f41cbbc6e3f437497c235098d3ee>

This appears to be in Golang Agent 1.13.x +

Looks like the setting is

`ELASTIC_APM_SERVER_CA_CERT_FILE`

So I think you you get this secret

`hm-apm-apm-http-ca-internal`

and the put this CA on the client / agent side and provide the fully qualified path to it in the `ELASTIC_APM_SERVER_CA_CERT_FILE` setting and then that should validate the Cert and CA

You will still need the `ELASTIC_APM_SERVER_CERT` setting

Give it a try and report back

Its is even mentioned in the 1.13.0 release notes [here](https://www.elastic.co/guide/en/apm/agent/go/current/release-notes-1.x.html#release-notes-1.13.0)

---

<div class="post-metadata">

**Author:** ![Hongbo-Miao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hongbo-miao/32/92664_2.png) [@Hongbo-Miao](https://discuss.elastic.co/u/Hongbo-Miao)\
**Post date:** [August 8, 2021, 7:26pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/7 "2021-08-08T19:26:13Z")

</div>

Hi @stephenb thank you so much for the help!

Sorry I missed this sentence early:

> The agent is going to need the apm-token as well that will be in `hm-apm-apm-token`

Here are 3 ways I tested that work:

First to get `ELASTIC_APM_SECRET_TOKEN`, run

```auto
kubectl get secret hm-apm-apm-token --namespace=elastic --output=go-template='{{index .data "secret-token"}}' | base64 -d && echo

```

## Method 1

Change to `ELASTIC_APM_VERIFY_SERVER_CERT: "false"` and remove `ELASTIC_APM_SERVER_CERT`, so like

```auto
ELASTIC_APM_SERVER_URL: "https://hm-apm-apm-http.elastic:8200"
ELASTIC_APM_ENVIRONMENT: "development"
ELASTIC_APM_LOG_LEVEL: "debug"
ELASTIC_APM_LOG_FILE: "stderr"
ELASTIC_APM_SECRET_TOKEN: "xxx"
ELASTIC_APM_VERIFY_SERVER_CERT: "false"

```

## Method 2

```auto
kubectl get secret hm-apm-apm-http-certs-public --namespace=elastic --output=go-template='{{index .data "tls.crt" | base64decode }}' > data/elastic-apm/tls.crt

```

Mounted **tls.crt** to the app, and pass these to Go app envs

```auto
ELASTIC_APM_SERVER_URL: "https://hm-apm-apm-http.elastic:8200"
ELASTIC_APM_ENVIRONMENT: "development"
ELASTIC_APM_LOG_LEVEL: "debug"
ELASTIC_APM_LOG_FILE: "stderr"
ELASTIC_APM_SECRET_TOKEN: "xxx"
ELASTIC_APM_SERVER_CERT: "/data/elastic-apm/tls.crt"

```

## Method 3

Adding `ELASTIC_APM_SERVER_CA_CERT_FILE` will still work:

```auto
kubectl get secret hm-apm-apm-http-certs-public --namespace=elastic --output=go-template='{{index .data "tls.crt" | base64decode }}' > data/elastic-apm/tls.crt
kubectl get secret hm-apm-apm-http-certs-public --namespace=elastic --output=go-template='{{index .data "ca.crt" | base64decode }}' > data/elastic-apm/ca.crt

```

Mounted both **tls.crt** and **ca.crt** to the app, and pass these to Go app envs

```auto
ELASTIC_APM_SERVER_URL: "https://hm-apm-apm-http.elastic:8200"
ELASTIC_APM_ENVIRONMENT: "development"
ELASTIC_APM_LOG_LEVEL: "debug"
ELASTIC_APM_LOG_FILE: "stderr"
ELASTIC_APM_SECRET_TOKEN: "xxx"
ELASTIC_APM_SERVER_CERT: "/data/elastic-apm/tls.crt"
ELASTIC_APM_SERVER_CA_CERT_FILE: "/data/elastic-apm/ca.crt"

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 8, 2021, 7:31pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/8 "2021-08-08T19:31:23Z")

</div>

Cool

Do all 3 methods work ?

I am surprised method 2 does without the CA... Perhaps the cert is full chain including the CA.

But good to know, glad you got it working!

---

<div class="post-metadata">

**Author:** ![Hongbo-Miao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hongbo-miao/32/92664_2.png) [@Hongbo-Miao](https://discuss.elastic.co/u/Hongbo-Miao)\
**Post date:** [August 8, 2021, 7:34pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/9 "2021-08-08T19:34:25Z")

</div>

Yeah, all work.

I had a closer look, for method 2 and method 3 using certificates, if without `ELASTIC_APM_SECRET_TOKEN `, it won't work.

When I am using certificate, am I supposed to use `ELASTIC_APM_SECRET_TOKEN` at same time?

If not, then it means my method 2 and 3 still do not work once removing `ELASTIC_APM_SECRET_TOKEN`. Both with get

> {"level":"error","time":"2021-08-08T19:40:40Z","message":"config request failed: request failed with 401 Unauthorized: {"error":"missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret\_token' or 'Authorization: ApiKey base64(API key ID:API key)'"}"}

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 8, 2021, 7:40pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/10 "2021-08-08T19:40:02Z")

</div>

Yes you **always** need the token.

That is a separate security check from the SSL certificate verification.

So I guess method 2 works because the tls.cert has the CA included... Full chain

You had this error at one point

"x509: certificate signed by unknown authority"

That usually means you need the CA, but good if you don't, perhaps earlier you pulled the wrong cert.

---

<div class="post-metadata">

**Author:** ![Hongbo-Miao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hongbo-miao/32/92664_2.png) [@Hongbo-Miao](https://discuss.elastic.co/u/Hongbo-Miao)\
**Post date:** [August 8, 2021, 7:41pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/11 "2021-08-08T19:41:59Z")

</div>

Cool, then all three work now! I don't know, I cannot reproduce that

> x509: certificate signed by unknown authority

any more even I tried my old config...

Anyway, thanks a lot!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 8, 2021, 7:43pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/12 "2021-08-08T19:43:15Z")

</div>

Good!

It was probably at bad cert at one point!

So method 2 it is...

This was a good conversation because some people will need the CA and that has been added now.

Good Luck!

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [August 9, 2021, 1:05am UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/13 "2021-08-09T01:05:52Z")

</div>

@stephenb good pickup on the new CA cert config. I've opened [docs: document ELASTIC\_APM\_SERVER\_CA\_CERT\_FILE · Issue #1001 · elastic/apm-agent-go · GitHub](https://github.com/elastic/apm-agent-go/issues/1001) to get that documented.

FWIW: method 2 works because it's checking apm-server's certificate matches exactly, rather than verifying the entire chain using the CA cert. i.e. "certificate pinning".

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 29, 2021, 9:06pm UTC](https://discuss.elastic.co/t/failed-to-verify-server-certificate-apm-go-agent-with-eck/280738/14 "2021-08-29T21:06:27Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.