Apmecho not sending anything to server

[Huynh_Nam]

Kibana version: 7.6.2

Elasticsearch version: 7.6.2

APM Server version: 7.6.2

APM Agent language and version: Golang (Echo) - go apm agent 1.7.2

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: helm with kubernetes https://github.com/elastic/helm-charts

Fresh install or upgraded from other version? Fresh install

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): my server does not sending anything to server, and there is no log output too

• Elasticsearch + Kibana + APM server was setup ok

• APM server log

2020-04-09T03:32:39.484Z	INFO	[request]	middleware/log_middleware.go:97	request ok	{"request_id": "69adfe0e-ae3e-44fd-9192-11ea0a4b518d", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "172.31.32.249", "user-agent": "kube-probe/1.15+", "response_code": 200}
2020-04-09T03:32:46.964Z	INFO	[request]	middleware/log_middleware.go:97	request ok	{"request_id": "60f10420-b303-453e-ab32-b1dc28eda835", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "172.31.32.249", "user-agent": "kube-probe/1.15+", "response_code": 200}
2020-04-09T03:32:49.484Z	INFO	[request]	middleware/log_middleware.go:97	request ok	{"request_id": "11e3a4a2-0002-4824-96c8-26edca6f3c13", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "172.31.32.249", "user-agent": "kube-probe/1.15+", "response_code": 200}
2020-04-09T03:32:56.964Z	INFO	[request]	middleware/log_middleware.go:97	request ok	{"request_id": "6945b05d-bd5e-45ab-bfb4-2dd7b428a161", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "172.31.32.249", "user-agent": "kube-probe/1.15+", "response_code": 200}
2020-04-09T03:32:59.484Z	INFO	[request]	middleware/log_middleware.go:97	request ok	{"request_id": "2b7fded3-d1ae-4998-ba6d-48537022bcbb", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "172.31.32.249", "user-agent": "kube-probe/1.15+", "response_code": 200}
2020-04-09T03:33:06.964Z	INFO	[request]	middleware/log_middleware.go:97	request ok	{"request_id": "43856eeb-b335-40dc-a7de-f831ed8b6294", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "172.31.32.249", "user-agent": "kube-probe/1.15+", "response_code": 200}
2020-04-09T03:33:09.484Z	INFO	[request]	middleware/log_middleware.go:97	request ok	{"request_id": "a83d0295-96bb-47b2-a7ff-fe117a1c1191", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "172.31.32.249", "user-agent": "kube-probe/1.15+", "response_code": 200}
2020-04-09T03:33:16.964Z	INFO	[request]	middleware/log_middleware.go:97	request ok	{"request_id": "63ced209-7e1b-4dc0-b247-42616c2a2950", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "172.31.32.249", "user-agent": "kube-probe/1.15+", "response_code": 200}

• Code

import(
	"go.elastic.co/apm/module/apmecho"
	// ...
)

func main() {
	os.Setenv("ELASTIC_APM_SERVER_URL", "http://<k8s-service-name>:8200")
	os.Setenv("ELASTIC_APM_API_KEY", "...")
	os.Setenv("ELASTIC_APM_SERVICE_NAME", "...")
	os.Setenv("ELASTIC_APM_SERVICE_NODE_NAME", "...")
	os.Setenv("ELASTIC_APM_HOSTNAME", "...")
	os.Setenv("ELASTIC_APM_ENVIRONMENT", "development")
	os.Setenv("ELASTIC_APM_LOG_FILE", dir+"/debug.log")
	os.Setenv("ELASTIC_APM_LOG_LEVEL", "debug")

	e := echo.New()
	e.Use(apmecho.Middleware())

	// ...

	// Start
	e.Logger.Fatal(e.Start(":3200"))
}

• Output

⇨ http server started on [::]:3200
2020-04-09T03:22:10Z | 172.31.36.157 | GET /api/notification/ping - 200 - 36.67µs
2020-04-09T03:22:12Z | 172.31.36.157 | GET /api/notification/ping - 200 - 40.251µs
2020-04-09T03:22:12Z | 172.31.36.157 | GET /api/notification/ping - 200 - 31.655µs
2020-04-09T03:22:13Z | 172.31.36.157 | GET /api/notification/ping - 200 - 31.117µs

I tried to call apis, but there's no log output from console, no log file in pod. And APM server did not receive any data from client.
Do I miss any steps to make it work?
Thank you!

[axw (Andrew Wilkins)]

The Go agent does not produce any logging output by default. You can enable logging by setting ELASTIC_APM_LOG_FILE and ELASTIC_APM_LOG_LEVEL. e.g. to log at debug level to stderr, set the environment variables ELASTIC_APM_LOG_FILE=stderr and ELASTIC_APM_LOG_LEVEL=debug.

This should give you some clues as to what the issue is.

[Huynh_Nam]

Hi @axw, thank for your help. When i put code like this

// ...
os.Setenv("ELASTIC_APM_LOG_FILE", "stderr")
os.Setenv("ELASTIC_APM_LOG_LEVEL", "debug")

then it not worked. But when i try to use k8s configmap, it's working now.
I don't know what's wrong when using code to set env, but thank you again!

[axw (Andrew Wilkins)]

I don't know what's wrong when using code to set env, but thank you again!

Ahh, sorry - I overlooked that you were setting environment variables in code before.

The problem with that approach is that the logger and global tracer are initialised before you can set environment variables inside the program. Setting them through a configmap is the right way to go.

There is an issue on GitHub to provide an API for re-initialising the logger, global tracer, etc., after setting environment variables. In case you would like to subscribe: Add a function to reinitialise all globals (tracer, transport, logger) · Issue #618 · elastic/apm-agent-go · GitHub