APM Dependencies tab is empty, but data present in Service map tab

[Offout (Ivan Filatov)]

Kibana version:
8.2.3

Elasticsearch version:
8.2.3

APM Server version:
8.2.3

APM Agent language and version:
PHP 1.5.2

Browser version:
Chrome 103, CLI

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Download page

Fresh install or upgraded from other version?
Fresh install

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

We are currently testing self-hosted APM server with PHP agent, and Service map is correct for all services (tab at main page) and for any individual service, but dependencies tab is completly empty in all services (depencencies tab at main page) and in every individual service.

Transactions are recordered from web requests and from CLI commands (cron, supervisord).

Steps to reproduce:

1. Run self-hosted APM server, PHP apps with APM agent
2. Go to Service map tab
3. Go to Dependencies tab

[lahsivjar (Vishal Raj)]

@Offout The dependencies tab is populated based on aggregated metrics (the service graph uses a different logic).

What is the result of metricset.name: "service_destination" AND span.destination.service.resource: * KQL for metrics-* index pattern?

Do you see any error in your APM-Server logs?

[Offout (Ivan Filatov)]

Thank you for your reply!
Search results metricset.name: "service_destination" AND span.destination.service.resource: * is empty.
As I can see metricset.name has only one value.
span.destination Is not present in "Avaliable fields"
I will share server logs a bit later

[lahsivjar (Vishal Raj)]

Search results metricset.name: "service_destination" AND span.destination.service.resource: * is empty.

This aligns with our observation of missing data in the APM dependencies tab. At this point, we can assume this to not be a UI issue and look into the APM Server log to find what is happening with the service destination aggregator.

[Offout (Ivan Filatov)]

Only errors I have found in logs are:

Invalid host param set: http://***.link:9200 http://***.link:9200, Error: parse "http://***:9200 http://***:9200": invalid character " " in host name

failed to index event (strict_dynamic_mapping_exception): mapping set to strict, dynamic introduction of [error] within [_doc] is not allowed

We are going to fix this two.

[lahsivjar (Vishal Raj)]

@Offout

failed to index event (strict_dynamic_mapping_exception): mapping set to strict, dynamic introduction of [error] within [_doc] is not allowed

Failure to index events might be the root cause since the internal indexes have strict mapping and the dependencies tab is based on service destination aggregation metrics produced by APM-Server. Let me know if fixing this error resolves issues with the dependencies tab.

[badhoprus (Grigorio Khmelnitsky)]

@lahsivjar

ot be a UI issue and look into the APM Server log to find what is happening with the service destination aggregator.

I have the same problem on my 8.3.3 Cloud installation. I recently update to 8.4.0 but it doesn`t help.

span.destination.service.resource: * isnt empy, but metricset.name: "service_destination"` empty.
I can see dependency it Trace -> Span Details:

And also on a service map, but dependency tab is empty.