APM Dependencies tab is empty, but data present in Service map tab

Elastic Observability APM
,
1

Kibana version:
8.2.3

Elasticsearch version:
8.2.3

APM Server version:
8.2.3

APM Agent language and version:
PHP 1.5.2

Browser version:
Chrome 103, CLI

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Download page

Fresh install or upgraded from other version?
Fresh install

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

We are currently testing self-hosted APM server with PHP agent, and Service map is correct for all services (tab at main page) and for any individual service, but dependencies tab is completly empty in all services (depencencies tab at main page) and in every individual service.

Transactions are recordered from web requests and from CLI commands (cron, supervisord).

service-map
service-map943×966 86.2 KB

dependencies
dependencies1288×674 79 KB

Steps to reproduce:

  1. Run self-hosted APM server, PHP apps with APM agent
  2. Go to Service map tab
  3. Go to Dependencies tab
2

@Offout The dependencies tab is populated based on aggregated metrics (the service graph uses a different logic).

What is the result of metricset.name: "service_destination" AND span.destination.service.resource: * KQL for metrics-* index pattern?

Do you see any error in your APM-Server logs?

3

Thank you for your reply!
Search results metricset.name: "service_destination" AND span.destination.service.resource: * is empty.
As I can see metricset.name has only one value.
span.destination Is not present in "Avaliable fields"
I will share server logs a bit later

image
image1292×365 27.4 KB

image

4

Search results metricset.name: "service_destination" AND span.destination.service.resource: * is empty.

This aligns with our observation of missing data in the APM dependencies tab. At this point, we can assume this to not be a UI issue and look into the APM Server log to find what is happening with the service destination aggregator.

5

Only errors I have found in logs are:

Invalid host param set: http://***.link:9200 http://***.link:9200, Error: parse "http://***:9200 http://***:9200": invalid character " " in host name

failed to index event (strict_dynamic_mapping_exception): mapping set to strict, dynamic introduction of [error] within [_doc] is not allowed

We are going to fix this two.

6

Failure to index events might be the root cause since the internal indexes have strict mapping and the dependencies tab is based on service destination aggregation metrics produced by APM-Server. Let me know if fixing this error resolves issues with the dependencies tab.

7

I have the same problem on my 8.3.3 Cloud installation. I recently update to 8.4.0 but it doesn`t help.

span.destination.service.resource: * isnt empy, but metricset.name: "service_destination"` empty.
I can see dependency it Trace -> Span Details:

image
image1234×452 59.1 KB

And also on a service map, but dependency tab is empty.

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.