APM Dependencies tab is empty, but data present in Service map tab

Kibana version:
8.2.3

Elasticsearch version:
8.2.3

APM Server version:
8.2.3

APM Agent language and version:
PHP 1.5.2

Browser version:
Chrome 103, CLI

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Download page

Fresh install or upgraded from other version?
Fresh install

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

We are currently testing self-hosted APM server with PHP agent, and Service map is correct for all services (tab at main page) and for any individual service, but dependencies tab is completly empty in all services (depencencies tab at main page) and in every individual service.

Transactions are recordered from web requests and from CLI commands (cron, supervisord).


Steps to reproduce:

  1. Run self-hosted APM server, PHP apps with APM agent
  2. Go to Service map tab
  3. Go to Dependencies tab

@Offout The dependencies tab is populated based on aggregated metrics (the service graph uses a different logic).

What is the result of metricset.name: "service_destination" AND span.destination.service.resource: * KQL for metrics-* index pattern?

Do you see any error in your APM-Server logs?

Thank you for your reply!
Search results metricset.name: "service_destination" AND span.destination.service.resource: * is empty.
As I can see metricset.name has only one value.
span.destination Is not present in "Avaliable fields"
I will share server logs a bit later


image

Search results metricset.name: "service_destination" AND span.destination.service.resource: * is empty.

This aligns with our observation of missing data in the APM dependencies tab. At this point, we can assume this to not be a UI issue and look into the APM Server log to find what is happening with the service destination aggregator.

Only errors I have found in logs are:

Invalid host param set: http://***.link:9200 http://***.link:9200, Error: parse "http://***:9200 http://***:9200": invalid character " " in host name
failed to index event (strict_dynamic_mapping_exception): mapping set to strict, dynamic introduction of [error] within [_doc] is not allowed

We are going to fix this two.

Failure to index events might be the root cause since the internal indexes have strict mapping and the dependencies tab is based on service destination aggregation metrics produced by APM-Server. Let me know if fixing this error resolves issues with the dependencies tab.