# APM trace view in Kibana displays the traced services as external services when they are called via the API gateway **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [June 21, 2024, 12:56pm UTC](https://discuss.elastic.co/t/apm-trace-view-in-kibana-displays-the-traced-services-as-external-services-when-they-are-called-via-the-api-gateway/361851 "2024-06-21T12:56:37Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![dacothe](https://avatars.discourse-cdn.com/v4/letter/d/dc4da7/32.png) [@dacothe](https://discuss.elastic.co/u/dacothe)\ **Post date:** [June 21, 2024, 12:56pm UTC](https://discuss.elastic.co/t/apm-trace-view-in-kibana-displays-the-traced-services-as-external-services-when-they-are-called-via-the-api-gateway/361851/1 "2024-06-21T12:56:37Z")
**Kibana version** : 8.6.2 **Elasticsearch version** : 8.6.2 **APM Server version** : 8.6.2 **APM Agent language and version** : Java, 1.30+ **Browser version** : Chrome 126.0.6478.115 Hello everyone, we use APM in our project to monitor our services in operation. When analysing the services, we noticed in the APM Trace View that the target service ('Service B') is displayed as an external component for the communication of the services via the API Gateway. ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/d/4df33ccffbe84b342e94bb65685cb1d50a3b6bca.png) In these cases, the call always follows the following pattern: `Service A` --\> `API Gateway` --\> `Service B` Distributed tracing is activated in the services `A` and `B` involved. The API gateway is not traced by the APM and is also an external system for our project. The trace correctly displays the calls of the various services in their sequence However, the type for service B is 'http' - I would expect `Service B` to be recognised here, as is the case with `Service A`. What criteria are used to set the service type in the trace? Shouldn't the call to 'Service B' via the API gateway be recognised as an internal call, since 'Service A' and 'Service B' are monitored in the same system via the APM agent? Thank you for your feedback! Best regards, Thilo ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [June 21, 2024, 1:24pm UTC](https://discuss.elastic.co/t/apm-trace-view-in-kibana-displays-the-traced-services-as-external-services-when-they-are-called-via-the-api-gateway/361851/2 "2024-06-21T13:24:29Z")
It should be the service.name field in the span. If that is absent then it has to fall back to the type of request ---
**Author:** ![Jonas\_Kunz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonas_kunz/32/110740_2.png) [@Jonas\_Kunz](https://discuss.elastic.co/u/Jonas_Kunz)\ **Post date:** [June 24, 2024, 10:35am UTC](https://discuss.elastic.co/t/apm-trace-view-in-kibana-displays-the-traced-services-as-external-services-when-they-are-called-via-the-api-gateway/361851/3 "2024-06-24T10:35:59Z")
I think the spans you marked with `ServiceA/...` are client spans. The fact that they don't have corresponding `Service B` transaction spans as children means distributed tracing is likely not correctly working in your setup. My guess here is that your API-Gateway does not preserve the [W3C trace context headers](https://www.w3.org/TR/trace-context/), making it impossible for elastic APM to correlate the transactions from `Service B` with the ones from `Service A`. ---
**Author:** ![dacothe](https://avatars.discourse-cdn.com/v4/letter/d/dc4da7/32.png) [@dacothe](https://discuss.elastic.co/u/dacothe)\ **Post date:** [June 24, 2024, 12:35pm UTC](https://discuss.elastic.co/t/apm-trace-view-in-kibana-displays-the-traced-services-as-external-services-when-they-are-called-via-the-api-gateway/361851/4 "2024-06-24T12:35:46Z")
Hello Jonas, thank you for your feedback - based on your reply, I realised that my image contained an error. Please find attached a corrected version of the image. This image shows that the transfer of header information via the API gateway works in principle – the spans of `Service B` are displayed within the transaction. ![APM_TraceView2](https://us1.discourse-cdn.com/elastic/original/3X/8/8/888eb693ad4aafc36170e0434f8aa78112fcbe41.png) Header Information `Service A`: ```auto http.request.headers.Traceparent: 00-4caa3c1fcd95dc020e8e19a9c9ef80e1-f966b2eca38df91a-01 http.request.headers.Tracestate: es=s:1 http.request.headers.X-B3-Parentspanid: f966b2eca38df91a http.request.headers.X-B3-Sampled: 1 http.request.headers.X-B3-Spanid: b142ed5182867fc5 http.request.headers.X-B3-Traceid: 4caa3c1fcd95dc020e8e19a9c9ef80e1 ``` Header Information `Service B`: ```auto http.request.headers.Traceparent: 00-4caa3c1fcd95dc020e8e19a9c9ef80e1-931e915089f36b75-01 http.request.headers.Tracestate: es=s:1 http.request.headers.X-B3-Parentspanid: 931e915089f36b75 http.request.headers.X-B3-Sampled: 1 http.request.headers.X-B3-Spanid: 2f8c704820fcbd2f http.request.headers.X-B3-Traceid: 4caa3c1fcd95dc020e8e19a9c9ef80e1 ``` However, the information that `Service B` is also monitored by the same APM server is obviously lost. If the call between `Service A` and `Service B` is made without using an API gateway, `Service B` is displayed as the type in the Kibana trace view. ![APM_TraceView3](https://us1.discourse-cdn.com/elastic/original/3X/6/0/60dd7a682b99b94422a21e6078a297e8b7858ffe.png) However, the information that `Service B` is also monitored by the same APM server is obviously lost. If the call between `Service A` and `Service B` is made without an API gateway, `Service B` is displayed as the type in the Kibana trace. I am confused why the display in Kibana is different - in the last case, the URL for calling the service is the difference. ---
**Author:** ![Jonas\_Kunz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonas_kunz/32/110740_2.png) [@Jonas\_Kunz](https://discuss.elastic.co/u/Jonas_Kunz)\ **Post date:** [June 25, 2024, 10:32am UTC](https://discuss.elastic.co/t/apm-trace-view-in-kibana-displays-the-traced-services-as-external-services-when-they-are-called-via-the-api-gateway/361851/5 "2024-06-25T10:32:42Z")
Do you know where those `X-B3` headers come from? The Elastic APM Agent does not set those. Looks like maybe some other tracing library is interfering and creating a span inbetween Service A and Service B. As a result, the parent of the Service B span is not known to elastic, causing it to not be rendered in the waterfall. Maybe your API gateway has some built-in tracing causing this behaviour? ---
**Author:** ![dacothe](https://avatars.discourse-cdn.com/v4/letter/d/dc4da7/32.png) [@dacothe](https://discuss.elastic.co/u/dacothe)\ **Post date:** [June 28, 2024, 1:34pm UTC](https://discuss.elastic.co/t/apm-trace-view-in-kibana-displays-the-traced-services-as-external-services-when-they-are-called-via-the-api-gateway/361851/6 "2024-06-28T13:34:01Z")
This header information is inserted into the request by the API gateway (KONG with ZipKing for tracing). The two `span.id` values in the X-B3 fields are not present in our ELK. This means that your explanation is correct – there is a span that is not part of the APM trace. I would expect the APM to ignore the X-B3 fields for tracing, since the W3C-compliant header information is also available. I also found this [issue](https://github.com/elastic/apm/issues/116) about it. For me, this means that the solution must be implemented in the API gateway: use traceparent / tracestate header. Thank you for your support!