# APM environment variables misconfigured

**URL:** <https://discuss.elastic.co/t/apm-environment-variables-misconfigured/200841>\
**Category:** APM\
**Tags:** go\
**Created:** [September 24, 2019, 10:46am UTC](https://discuss.elastic.co/t/apm-environment-variables-misconfigured/200841 "2019-09-24T10:46:58Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![alexclifford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexclifford/32/54957_2.png) [@alexclifford](https://discuss.elastic.co/u/alexclifford)\
**Post date:** [September 24, 2019, 10:46am UTC](https://discuss.elastic.co/t/apm-environment-variables-misconfigured/200841/1 "2019-09-24T10:46:58Z")

</div>

**Kibana version** : 7.3.2

**Elasticsearch version** : 7.3.2

**APM Server version** : 7.3.2

**APM Agent language and version** :  
[go.elastic.co/apm](http://go.elastic.co/apm) v1.5.0  
[go.elastic.co/apm/module/apmecho](http://go.elastic.co/apm/module/apmecho) v1.5.0  
[go.elastic.co/apm/module/apmsql](http://go.elastic.co/apm/module/apmsql) v1.5.0

**Browser version** : N/A

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: Elastic Cloud

**Fresh install or upgraded from other version?** N/A

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

I have a Kubernetes Pod that is pushing to Elastic Cloud via the Go APM echo module. It has the following settings:

```auto
⇒ kc -n pg-10 exec -it api-v3-5c77c77db6-xh7sx sh
/app # env | grep ELASTIC
ELASTIC_APM_ENVIRONMENT=pg-10
ELASTIC_APM_SECRET_TOKEN=<redacted>
ELASTIC_APM_IGNORE_URLS=/metrics
ELASTIC_APM_SERVICE_NAME=api-v3
ELASTIC_APM_SERVER_URL=<redacted>

```

In Kibana I'm still seeing /metrics calls, see the image I've saved at [https://i.imgur.com/8ijkjHQ.png](https://i.imgur.com/8ijkjHQ.png)

How do I go about also excluding all "/metrics" and "/" routes. But also I'm interested in excluding all metric data completely, however I tried ELASTIC\_APM\_METRICS\_INTERVAL=0s but the metrics index is still receiving data.

I've also tried all all of these environment variables also but ENVIRONMENT and SERVICE\_NAME seem to be the only variables affecting anything:

```auto
ELASTIC_APM_METRICS_INTERVAL=0s
ELASTIC_APM_DISABLE_METRICS=*
ELASTIC_APM_CAPTURE_HEADERS=false
ELASTIC_APM_BREAKDOWN_METRICS=false
ELASTIC_APM_IGNORE_URLS=/metrics, /
ELASTIC_APM_STACK_TRACE_LIMIT=0

```

Appreciate any insight into how to configure this correctly.

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [September 25, 2019, 6:05am UTC](https://discuss.elastic.co/t/apm-environment-variables-misconfigured/200841/2 "2019-09-25T06:05:58Z")

</div>

Hi @alexclifford, welcome to the forum!

This is an odd one. I've just confirmed that the environment variables are honoured in a simple Echo application:

```auto
package main

import (
        "github.com/labstack/echo"
        "go.elastic.co/apm/module/apmecho"
)

func main() {
        e := echo.New()
        e.Use(apmecho.Middleware())
        e.GET("/", func(c echo.Context) error { return nil })
        e.GET("/metrics", func(c echo.Context) error { return nil })
        e.Logger.Fatal(e.Start(":8080"))
}

```

Are you passing any options into `apmecho.Middleware()`?

Setting `ELASTIC_APM_METRICS_INTERVAL=0s` correctly disables the metrics, and `ELASTIC_APM_IGNORE_URLS=/metrics` filters out transactions with matching URL path.

Are you defining the environment variables through the pod spec? Can you confirm that the server process has picked up the env vars? e.g. by running

```auto
cat /proc/`pidof "your program name"`/environ | tr '\0' '\n' | grep ELASTIC

```

---

<div class="post-metadata">

**Author:** ![alexclifford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexclifford/32/54957_2.png) [@alexclifford](https://discuss.elastic.co/u/alexclifford)\
**Post date:** [September 25, 2019, 10:28am UTC](https://discuss.elastic.co/t/apm-environment-variables-misconfigured/200841/3 "2019-09-25T10:28:03Z")

</div>

Hi @axw thanks for testing this and your input, it has helped point me in the right direction.

I can see in the Kubernetes Pod that the process running does have all the correct environment variables available to it, at least at the time of running viewing the proc environ file.  
But something is definitely at play here as it still refuses to honour them... some of the time! One namespace/environment running the process managed to pick them up and is working as expected. Identical namespaces just with a different context.service.environment value is in the same predicament. Restarting Pods doesn't seem to fix them.

I've been able to test locally with the same environment variables and there is never an issue so it seems like a race condition on the Kubernetes Pods where Echo is loading before the environment variables are available? Quite bizarre.

I will keep debugging Kubernetes to see if I can find a solution, thanks again for your help and suggestions.

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [September 26, 2019, 2:26am UTC](https://discuss.elastic.co/t/apm-environment-variables-misconfigured/200841/4 "2019-09-26T02:26:25Z")

</div>

Bizarre indeed!

Are you passing environment through the kubernetes pod spec/container definition? Or are you doing something else, like loading the environment from a file? The `ELASTIC_APM_*` environment variables are evaluated at program initialisation time, so if the environment variables aren't set when the process is first created, that might be the issue.

If you find out the cause, I'll be interested to hear back.

---

<div class="post-metadata">

**Author:** ![alexclifford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexclifford/32/54957_2.png) [@alexclifford](https://discuss.elastic.co/u/alexclifford)\
**Post date:** [October 13, 2019, 11:50am UTC](https://discuss.elastic.co/t/apm-environment-variables-misconfigured/200841/5 "2019-10-13T11:50:38Z")

</div>

I'm unsure what the issue was. I removed all of our application namespaces in Kubernetesand re-created them pointing at a new Elastic Cloud deployment and the issue was no longer there. I'm guessing it was something on our Kubernetes side.

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 3, 2019, 7:50am UTC](https://discuss.elastic.co/t/apm-environment-variables-misconfigured/200841/6 "2019-11-03T07:50:40Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.