Kibana version:
7.6.1
Elasticsearch version:
7.6.1
APM Server version:
7.6.1
APM Agent language and version:
traefik(go agent), 2.4.7
Browser version:
N/A
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
from docker
Fresh install or upgraded from other version?
no
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
currently using Logstash, ingress port
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
traefik settings:
- --tracing.elastic.serverUrl=http://xx.xxx.x.xx:xxxx
- --tracing.elastic.serviceEnvironment=dev
Expected to send transaction and span data through apm-server to logstash(final destination is elasticsearch). Apm-server successfully get data from other apm agent(python) and send it to logstash but from traefik(go agent), it only delivers metric datas(i can see data at kibana-apm tab)
APM server shows error log like below(data validation error) and it shows transaction and span data are sent from traefik but the validation is wrong.
I searched there was similar bug with using dotnet tho, not sure if this is a bug
Steps to reproduce:
1.
2.
3.
Provide logs and/or server output (if relevant):
2021-03-17T09:15:41.665Z ERROR [request] middleware/log_middleware.go:95 data validation error {"request_id": "f2350436-eae7-44a3-b9f4-dcb8d314914c", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.255.202.151", "user-agent": "elasticapm-go/1.7.0 go/go1.15.7", "response_code": 400, "error": "error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n I[#] S[#/allOf/3] allOf failed\n I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.067705,\"id\":\"2a4bb5df0ab76f06\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972480966543,\"trace_id\":\"6c6c4a2dc29dc2d4ea7f06006c9a5aef\",\"transaction_id\":\"6c6c4a2dc29dc2d4\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"service_name\":\"api@internal\",\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\",\"span_kind\":\"client\"}},\"parent_id\":\"6c6c4a2dc29dc2d4\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n I[#] S[#/allOf/3] allOf failed\n I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.070902,\"id\":\"af11a3515ad992cc\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972481967112,\"trace_id\":\"97e67eed9b7a4c5a7dca1f7f7f15b54a\",\"transaction_id\":\"97e67eed9b7a4c5a\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\",\"span_kind\":\"client\",\"service_name\":\"api@internal\",\"router_name\":\"api@internal\"}},\"parent_id\":\"97e67eed9b7a4c5a\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n I[#] S[#/allOf/3] allOf failed\n I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.06988,\"id\":\"97034eb5ebacea0c\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972481972415,\"trace_id\":\"f9546745bf9d4617d9b6acb53193dff2\",\"transaction_id\":\"f9546745bf9d4617\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/tcp/routers?search=\\u0026status=\\u0026per_page=10\\u0026page=1\"},\"tags\":{\"span_kind\":\"client\",\"service_name\":\"api@internal\",\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\"}},\"parent_id\":\"f9546745bf9d4617\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n I[#] S[#/allOf/3] allOf failed\n I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.06521199999999999,\"id\":\"0678c5a7b37e0ac0\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972485966418,\"trace_id\":\"743825f218e6983fbc56bdf77cc751ed\",\"transaction_id\":\"743825f218e6983f\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"http_status_code\":\"200\",\"span_kind\":\"client\",\"service_name\":\"api@internal\",\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\"}},\"parent_id\":\"743825f218e6983f\",\"stacktrace\":[],\"subtype\":\"http\"}}], error validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n I[#] S[#/allOf/3] allOf failed\n I[#/context/destination/service] S[#/allOf/3/properties/context/properties/destination/properties/service/required] missing properties: \"name\", \"resource\" [{\"span\":{\"duration\":0.058368,\"id\":\"93872e5c3408271a\",\"name\":\"forward api@internal/api@internal\",\"timestamp\":1615972486968100,\"trace_id\":\"368f99fae6360c007a0d5718eadd6b8f\",\"transaction_id\":\"368f99fae6360c00\",\"type\":\"external\",\"context\":{\"destination\":{\"service\":{\"type\":\"external\"}},\"http\":{\"url\":\"http:///api/overview\"},\"tags\":{\"router_name\":\"api@internal\",\"http_host\":\"202.8.174.146:5000\",\"http_status_code\":\"200\",\"span_kind\":\"client\",\"service_name\":\"api@internal\"}},\"parent_id\":\"368f99fae6360c00\",\"stacktrace\":[],\"subtype\":\"http\"}}]"}