# Troubleshooting missing links for some OpenTelemetry services in the APM Service Map **URL:** \ **Category:** APM\ **Tags:** open-telemetry\ **Created:** [April 14, 2022, 4:13pm UTC](https://discuss.elastic.co/t/troubleshooting-missing-links-for-some-opentelemetry-services-in-the-apm-service-map/302451 "2022-04-14T16:13:42Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![davemoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davemoore/32/134233_2.png) [@davemoore](https://discuss.elastic.co/u/davemoore)\ **Post date:** [April 14, 2022, 4:13pm UTC](https://discuss.elastic.co/t/troubleshooting-missing-links-for-some-opentelemetry-services-in-the-apm-service-map/302451/1 "2022-04-14T16:13:42Z")
**Observed behavior:** The APM service map doesn't display links between some OpenTelemetry services, but those links do appear in the trace samples. When searching in Discover for the trace.id of one of the orphaned services on the service map (e.g. payment service), spans _are_ included for other services (e.g. api-gateway service), which is expected and desired. When loading the Overview page for one of the orphaned services on the service map (e.g. payment service) Kibana displays a popup with the error message: ```auto Error while fetching resource Error search_phase_execution_exception: [script_exception] Reason: link error (500) ``` Similarly, some but not all distributed traces are captured despite nearly identical instrumentation code among the services. **Question:** What is the best way to troubleshoot missing links in the service map? What does the service map require to display links between services instrumented with OpenTelemetry? **Environment:** - Elastic Cloud on GCP us-east1 with Elastic 8.1.2 - All services are instrumented with OpenTelemetry (see [source code](https://github.com/microbs-io/microbs/tree/main/apps/ecommerce/services)) - All services use Python, except for api-gateway which uses nginx (via the opentelemetry [module](https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/nginx)) - The Python services use the same code and environment variables for instrumentation - The screenshots (below) were scoped to the same time ranges and environments **Screenshot 1 of 2** - The service map is missing links among some services (the product and payment service). ![service-map](https://us1.discourse-cdn.com/elastic/original/3X/e/5/e50fb5559e82edb8dd8a2f74fe54cb788fbdbab2.jpeg) **Screenshot 2 of 2** - This trace sample does display links between the services that were unlinked in the service map. This screenshot shows that the payment service is linked to the api-gateway service, but that link doesn't appear in the service map. ![trace](https://us1.discourse-cdn.com/elastic/original/3X/7/b/7b92b1d181618987a868181dad30a7c99bdacf44.jpeg) ---
**Author:** ![davemoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davemoore/32/134233_2.png) [@davemoore](https://discuss.elastic.co/u/davemoore)\ **Post date:** [April 14, 2022, 6:27pm UTC](https://discuss.elastic.co/t/troubleshooting-missing-links-for-some-opentelemetry-services-in-the-apm-service-map/302451/2 "2022-04-14T18:27:43Z")
I looked at the JSON response from the XHR that the service map sends to `/internal/apm/service-map` and I see that `service.environment` is `null` for the two orphaned services (payment and products). I'm unsure why those two services have `null` values because they are given the same environment variables (`OTEL_RESOURCE_ATTRIBUTES=deployment.environment=development`) and instrumentation logic as the other services. This does seem like something that would cause the services to appear orphaned in the service map. EDIT: I've confirmed that the spans for payment and product services all have `service.deployment` set to `development`, verified by searching Discover with the query `(service.name:payment OR service.name:product) AND NOT service.environment:development)`. I'm not sure why these services have `null` as the `service.environment` in the service map. ```auto { "elements": [{ "data": { "id": "web-gateway", "service.environment": "development", "service.name": "web-gateway", "agent.name": "opentelemetry/python" } }, { "data": { "id": "api-gateway", "service.name": "api-gateway", "agent.name": "opentelemetry/cpp" } }, { "data": { "id": "content", "service.environment": "development", "service.name": "content", "agent.name": "opentelemetry/python" } }, { "data": { "span.subtype": "http", "span.destination.service.resource": "storage.googleapis.com:443", "span.type": "external", "id": ">storage.googleapis.com:443", "label": "storage.googleapis.com:443" } }, { "data": { "id": "checkout", "service.environment": "development", "service.name": "checkout", "agent.name": "opentelemetry/python" } }, { "data": { "id": "cart", "service.environment": "development", "service.name": "cart", "agent.name": "opentelemetry/python" } }, { "data": { "span.subtype": "redis", "span.destination.service.resource": "redis", "span.type": "db", "id": ">redis", "label": "redis" } }, { "data": { "service.name": "product", "agent.name": "opentelemetry/python", "service.environment": null, "id": "product" } }, { "data": { "service.name": "payment", "agent.name": "opentelemetry/python", "service.environment": null, "id": "payment" } }, { "data": { "source": "api-gateway", "target": "cart", "id": "api-gateway~cart", "sourceData": { "id": "api-gateway", "service.name": "api-gateway", "agent.name": "opentelemetry/cpp" }, "targetData": { "id": "cart", "service.environment": "development", "service.name": "cart", "agent.name": "opentelemetry/python" } } }, { "data": { "source": "api-gateway", "target": "checkout", "id": "api-gateway~checkout", "sourceData": { "id": "api-gateway", "service.name": "api-gateway", "agent.name": "opentelemetry/cpp" }, "targetData": { "id": "checkout", "service.environment": "development", "service.name": "checkout", "agent.name": "opentelemetry/python" }, "bidirectional": true } }, { "data": { "source": "api-gateway", "target": "content", "id": "api-gateway~content", "sourceData": { "id": "api-gateway", "service.name": "api-gateway", "agent.name": "opentelemetry/cpp" }, "targetData": { "id": "content", "service.environment": "development", "service.name": "content", "agent.name": "opentelemetry/python" } } }, { "data": { "source": "cart", "target": ">redis", "id": "cart~>redis", "sourceData": { "id": "cart", "service.environment": "development", "service.name": "cart", "agent.name": "opentelemetry/python" }, "targetData": { "span.subtype": "redis", "span.destination.service.resource": "redis", "span.type": "db", "id": ">redis", "label": "redis" } } }, { "data": { "source": "checkout", "target": "api-gateway", "id": "checkout~api-gateway", "sourceData": { "id": "checkout", "service.environment": "development", "service.name": "checkout", "agent.name": "opentelemetry/python" }, "targetData": { "id": "api-gateway", "service.name": "api-gateway", "agent.name": "opentelemetry/cpp" }, "isInverseEdge": true } }, { "data": { "source": "content", "target": ">storage.googleapis.com:443", "id": "content~>storage.googleapis.com:443", "sourceData": { "id": "content", "service.environment": "development", "service.name": "content", "agent.name": "opentelemetry/python" }, "targetData": { "span.subtype": "http", "span.destination.service.resource": "storage.googleapis.com:443", "span.type": "external", "id": ">storage.googleapis.com:443", "label": "storage.googleapis.com:443" } } }, { "data": { "source": "web-gateway", "target": "api-gateway", "id": "web-gateway~api-gateway", "sourceData": { "id": "web-gateway", "service.environment": "development", "service.name": "web-gateway", "agent.name": "opentelemetry/python" }, "targetData": { "id": "api-gateway", "service.name": "api-gateway", "agent.name": "opentelemetry/cpp" } } }] } ``` ---
**Author:** ![davemoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davemoore/32/134233_2.png) [@davemoore](https://discuss.elastic.co/u/davemoore)\ **Post date:** [April 18, 2022, 3:32pm UTC](https://discuss.elastic.co/t/troubleshooting-missing-links-for-some-opentelemetry-services-in-the-apm-service-map/302451/3 "2022-04-18T15:32:00Z")
I've moved this conversation to Github (elastic/kibana [130319](https://github.com/elastic/kibana/issues/130319)) as a possible bug. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 9, 2022, 11:32am UTC](https://discuss.elastic.co/t/troubleshooting-missing-links-for-some-opentelemetry-services-in-the-apm-service-map/302451/4 "2022-05-09T11:32:35Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.