# Data validation error from traefik(go agent) to elastic apm server **URL:** \ **Category:** APM\ **Tags:** server\ **Created:** [March 18, 2021, 7:47am UTC](https://discuss.elastic.co/t/data-validation-error-from-traefik-go-agent-to-elastic-apm-server/267614 "2021-03-18T07:47:40Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![jinner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jinner/32/48265_2.png) [@jinner](https://discuss.elastic.co/u/jinner)\ **Post date:** [March 18, 2021, 7:47am UTC](https://discuss.elastic.co/t/data-validation-error-from-traefik-go-agent-to-elastic-apm-server/267614/1 "2021-03-18T07:47:40Z")
**Kibana version** : 7.6.1 **Elasticsearch version** : 7.6.1 **APM Server version** : 7.6.1 **APM Agent language and version** : traefik(go agent), 2.4.7 **Browser version** : N/A **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: from docker **Fresh install or upgraded from other version?** no **Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. currently using Logstash, ingress port **Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**: ```auto traefik settings: - --tracing.elastic.serverUrl=http://xx.xxx.x.xx:xxxx - --tracing.elastic.serviceEnvironment=dev ``` Expected to send transaction and span data through apm-server to logstash(final destination is elasticsearch). Apm-server successfully get data from other apm agent(python) and send it to logstash but from traefik(go agent), it only delivers metric datas(i can see data at kibana-apm tab) APM server shows error log like below(data validation error) and it shows transaction and span data are sent from traefik but the validation is wrong. I searched there was similar bug with using dotnet tho, not sure if this is a bug **Steps to reproduce** : 1. 2. 3. **Provide logs and/or server output (if relevant)**: ```auto 2021-03-17T09:15:41.665Z ERROR [request] middleware/log_middleware.go:95 data validation error {"request_id": "f2350436-eae7-44a3-b9f4-dcb8d314914c", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.255.202.151", "user-agent": "elasticapm-go/1.7.0 go/go1.15.7", "response_code": 400, "error": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n I[#] S[#/allOf/3] allOf failed\n I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.067705,\"id\":\"2a4bb5df0ab76f06\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972480966543,\"trace_id\":\"6c6c4a2dc29dc2d4ea7f06006c9a5aef\",\"transaction_id\":\"6c6c4a2dc29dc2d4\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"service_name\":\"api@internal\",\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\",\"span_kind\":\"client\"}},\"parent_id\":\"6c6c4a2dc29dc2d4\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n I[#] S[#/allOf/3] allOf failed\n I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.070902,\"id\":\"af11a3515ad992cc\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972481967112,\"trace_id\":\"97e67eed9b7a4c5a7dca1f7f7f15b54a\",\"transaction_id\":\"97e67eed9b7a4c5a\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\",\"span_kind\":\"client\",\"service_name\":\"api@internal\",\"router_name\":\"api@internal\"}},\"parent_id\":\"97e67eed9b7a4c5a\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n I[#] S[#/allOf/3] allOf failed\n I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.06988,\"id\":\"97034eb5ebacea0c\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972481972415,\"trace_id\":\"f9546745bf9d4617d9b6acb53193dff2\",\"transaction_id\":\"f9546745bf9d4617\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/tcp/routers?search=\\u0026status=\\u0026per_page=10\\u0026page=1\"},\"tags\":{\"span_kind\":\"client\",\"service_name\":\"api@internal\",\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\"}},\"parent_id\":\"f9546745bf9d4617\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n I[#] S[#/allOf/3] allOf failed\n I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.06521199999999999,\"id\":\"0678c5a7b37e0ac0\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972485966418,\"trace_id\":\"743825f218e6983fbc56bdf77cc751ed\",\"transaction_id\":\"743825f218e6983f\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"http_status_code\":\"200\",\"span_kind\":\"client\",\"service_name\":\"api@internal\",\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\"}},\"parent_id\":\"743825f218e6983f\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n I[#] S[#/allOf/3] allOf failed\n I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.058368,\"id\":\"93872e5c3408271a\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972486968100,\"trace_id\":\"368f99fae6360c007a0d5718eadd6b8f\",\"transaction_id\":\"368f99fae6360c00\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\",\"span_kind\":\"client\",\"service_name\":\"api@internal\"}},\"parent_id\":\"368f99fae6360c00\",\"stacktrace\":[],\"subtype\":\"http\"}}]"} ``` ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [March 18, 2021, 8:43am UTC](https://discuss.elastic.co/t/data-validation-error-from-traefik-go-agent-to-elastic-apm-server/267614/2 "2021-03-18T08:43:18Z")
Hi @jinner, thanks for raising this. The Go agent should not be causing these data validation errors, so that's a bug. It looks like the bug is caused by Traefik not quite following the OpenTracing [Semantic conventions](https://opentracing.io/specification/conventions/). The `http.url` tag it's sending does not contain the host name, and that is being sent in a non-standard `http.host` tag. We'll fix the data validation issue (noted in [[META 424] Public API extended by destination fields · Issue #916 · elastic/apm-agent-go · GitHub](https://github.com/elastic/apm-agent-go/issues/916)), and I've opened [apmot: handle http.host tag · Issue #917 · elastic/apm-agent-go · GitHub](https://github.com/elastic/apm-agent-go/issues/917) to support `http.host`. ---
**Author:** ![jinner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jinner/32/48265_2.png) [@jinner](https://discuss.elastic.co/u/jinner)\ **Post date:** [March 18, 2021, 1:56pm UTC](https://discuss.elastic.co/t/data-validation-error-from-traefik-go-agent-to-elastic-apm-server/267614/3 "2021-03-18T13:56:59Z")
@awx Thanks for quick reply. If i understood it right, there would be two solutions. One is wait for the fix that es-server doesn't send full context(or http context) in case of not following OpenTracing Semantic convetions. The other that you recommended ([apmot: handle http.host tag · Issue #917 · elastic/apm-agent-go · GitHub](https://github.com/elastic/apm-agent-go/issues/917)) is fixing data that traefik sends by adding http.host field to http.url or remove http.host(?). According to the semantic conventions, http.url is standard URI format. So in this case, how do i add http.host field to http.url? Is it okay to append hostname at the end of the url? just curious. ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [March 22, 2021, 3:09am UTC](https://discuss.elastic.co/t/data-validation-error-from-traefik-go-agent-to-elastic-apm-server/267614/4 "2021-03-22T03:09:44Z")
> If i understood it right, there would be two solutions. There's two parts to the complete solution. > One is wait for the fix that es-server doesn't send full context(or http context) in case of not following OpenTracing Semantic convetions. Right. So if we fix this bit, the agent won't send invalid payloads to APM Server. The payloads will be a little bit incomplete though -- events won't have complete, valid URLs, and services won't show up in the service map. > The other that you recommended ([apmot: handle http.host tag · Issue #917 · elastic/apm-agent-go · GitHub](https://github.com/elastic/apm-agent-go/issues/917)) is fixing data that traefik sends by adding http.host field to http.url or remove http.host(?). According to the semantic conventions, http.url is standard URI format. So in this case, how do i add http.host field to http.url? Is it okay to append hostname at the end of the url? just curious. From the error message you pasted, Traefik is setting `http.url` to "http:///api/overview" (for example). We'll need to parse this, and insert the value of `http.host` in after "http://" so we end up with "[http://host](http://host):port/api/overview". This will need to be handled somewhere around [https://github.com/elastic/apm-agent-go/blob/df272e446c4a3eebd92910efe1c572fe7ee47be3/module/apmot/span.go#L170-L172](https://github.com/elastic/apm-agent-go/blob/df272e446c4a3eebd92910efe1c572fe7ee47be3/module/apmot/span.go#L170-L172). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 11, 2021, 11:10pm UTC](https://discuss.elastic.co/t/data-validation-error-from-traefik-go-agent-to-elastic-apm-server/267614/5 "2021-04-11T23:10:24Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.