Data validation error from traefik(go agent) to elastic apm server

[jinner (Seongjin Yoon)]

Kibana version:
7.6.1
Elasticsearch version:
7.6.1
APM Server version:
7.6.1
APM Agent language and version:
traefik(go agent), 2.4.7
Browser version:
N/A
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
from docker
Fresh install or upgraded from other version?
no
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

currently using Logstash, ingress port

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

traefik settings:
- --tracing.elastic.serverUrl=http://xx.xxx.x.xx:xxxx
- --tracing.elastic.serviceEnvironment=dev

Expected to send transaction and span data through apm-server to logstash(final destination is elasticsearch). Apm-server successfully get data from other apm agent(python) and send it to logstash but from traefik(go agent), it only delivers metric datas(i can see data at kibana-apm tab)
APM server shows error log like below(data validation error) and it shows transaction and span data are sent from traefik but the validation is wrong.
I searched there was similar bug with using dotnet tho, not sure if this is a bug

Steps to reproduce:
1.
2.
3.

Provide logs and/or server output (if relevant):

2021-03-17T09:15:41.665Z	ERROR	[request]	middleware/log_middleware.go:95	data validation error	{"request_id": "f2350436-eae7-44a3-b9f4-dcb8d314914c", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.255.202.151", "user-agent": "elasticapm-go/1.7.0 go/go1.15.7", "response_code": 400, "error": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n  I[#] S[#/allOf/3] allOf failed\n    I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.067705,\"id\":\"2a4bb5df0ab76f06\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972480966543,\"trace_id\":\"6c6c4a2dc29dc2d4ea7f06006c9a5aef\",\"transaction_id\":\"6c6c4a2dc29dc2d4\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"service_name\":\"api@internal\",\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\",\"span_kind\":\"client\"}},\"parent_id\":\"6c6c4a2dc29dc2d4\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n  I[#] S[#/allOf/3] allOf failed\n    I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.070902,\"id\":\"af11a3515ad992cc\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972481967112,\"trace_id\":\"97e67eed9b7a4c5a7dca1f7f7f15b54a\",\"transaction_id\":\"97e67eed9b7a4c5a\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\",\"span_kind\":\"client\",\"service_name\":\"api@internal\",\"router_name\":\"api@internal\"}},\"parent_id\":\"97e67eed9b7a4c5a\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n  I[#] S[#/allOf/3] allOf failed\n    I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.06988,\"id\":\"97034eb5ebacea0c\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972481972415,\"trace_id\":\"f9546745bf9d4617d9b6acb53193dff2\",\"transaction_id\":\"f9546745bf9d4617\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/tcp/routers?search=\\u0026status=\\u0026per_page=10\\u0026page=1\"},\"tags\":{\"span_kind\":\"client\",\"service_name\":\"api@internal\",\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\"}},\"parent_id\":\"f9546745bf9d4617\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n  I[#] S[#/allOf/3] allOf failed\n    I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.06521199999999999,\"id\":\"0678c5a7b37e0ac0\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972485966418,\"trace_id\":\"743825f218e6983fbc56bdf77cc751ed\",\"transaction_id\":\"743825f218e6983f\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"http_status_code\":\"200\",\"span_kind\":\"client\",\"service_name\":\"api@internal\",\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\"}},\"parent_id\":\"743825f218e6983f\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n  I[#] S[#/allOf/3] allOf failed\n    I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.058368,\"id\":\"93872e5c3408271a\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972486968100,\"trace_id\":\"368f99fae6360c007a0d5718eadd6b8f\",\"transaction_id\":\"368f99fae6360c00\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\",\"span_kind\":\"client\",\"service_name\":\"api@internal\"}},\"parent_id\":\"368f99fae6360c00\",\"stacktrace\":[],\"subtype\":\"http\"}}]"}

[axw (Andrew Wilkins)]

Hi @jinner, thanks for raising this.

The Go agent should not be causing these data validation errors, so that's a bug. It looks like the bug is caused by Traefik not quite following the OpenTracing Semantic conventions. The http.url tag it's sending does not contain the host name, and that is being sent in a non-standard http.host tag.

We'll fix the data validation issue (noted in [META 424] Public API extended by destination fields · Issue #916 · elastic/apm-agent-go · GitHub), and I've opened apmot: handle http.host tag · Issue #917 · elastic/apm-agent-go · GitHub to support http.host.

[jinner (Seongjin Yoon)]

@awx Thanks for quick reply. If i understood it right, there would be two solutions. One is wait for the fix that es-server doesn't send full context(or http context) in case of not following OpenTracing Semantic convetions. The other that you recommended (apmot: handle http.host tag · Issue #917 · elastic/apm-agent-go · GitHub) is fixing data that traefik sends by adding http.host field to http.url or remove http.host(?). According to the semantic conventions, http.url is standard URI format. So in this case, how do i add http.host field to http.url? Is it okay to append hostname at the end of the url? just curious.

[axw (Andrew Wilkins)]

If i understood it right, there would be two solutions.

There's two parts to the complete solution.

One is wait for the fix that es-server doesn't send full context(or http context) in case of not following OpenTracing Semantic convetions.

Right. So if we fix this bit, the agent won't send invalid payloads to APM Server. The payloads will be a little bit incomplete though -- events won't have complete, valid URLs, and services won't show up in the service map.

The other that you recommended (apmot: handle http.host tag · Issue #917 · elastic/apm-agent-go · GitHub) is fixing data that traefik sends by adding http.host field to http.url or remove http.host(?). According to the semantic conventions, http.url is standard URI format. So in this case, how do i add http.host field to http.url? Is it okay to append hostname at the end of the url? just curious.

From the error message you pasted, Traefik is setting http.url to "http:///api/overview" (for example). We'll need to parse this, and insert the value of http.host in after "http://" so we end up with "http://host:port/api/overview".

This will need to be handled somewhere around https://github.com/elastic/apm-agent-go/blob/df272e446c4a3eebd92910efe1c572fe7ee47be3/module/apmot/span.go#L170-L172.