Node Elastic APM agent. Transactions missing trace samples, timeline and metadata

[Abror_Abdullaev]

The environment:
Kibana: v 8.8.2
APM Agent elastic-apm-node v4.0.0
NetstJs Framework

I am starting the agent at the beginning of my application

//main.ts

import * as dotenv from 'dotenv';
import * as apm from 'elastic-apm-node';

dotenv.config();
apm.start({
  serverUrl: process.env.APM_SERVER_URL,
  serviceName: process.env.APM_SERVICE_NAME,
  environment: process.env.APM_ENVIRONMENT,
  verifyServerCert: !!process.env.APM_VERIFY_SERVER_CERT,
  captureBody: 'all',
  logLevel: 'debug',
});

...
...

boostrap();

In Kibana I can see all my requests as transactions, they appear after less than a minute after request. The name of the transaction is the url of the request, so its correct. But if I open the transaction to inspect, it is infinitly loading in the section where it should show the trace samples, timeline and metadata, also it doesn't show the response code.

Though I am aware the apm-agent doesn't support nestjs well. I tried to manually end the transaction after requests, tried to crash the app and different ways to understand why the traces and metadata is not visible. In the browser, in Kibana I see no traces are retrieved. I guess they are just not sent from the agent to elastic.

Also here are the debug logs on simple healthcheck url that simply returns 200 status code.

{"log.level":"debug","@timestamp":"2023-09-18T14:47:46.432Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"intercepted request event call to http.Server.prototype.emit for /"}
{"log.level":"debug","@timestamp":"2023-09-18T14:47:46.433Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"start trace {\"trans\":\"60a6daf2e7e5ed2e\",\"trace\":\"1ba8bd8932395bf36c08fd7da8f8ff60\",\"name\":\"unnamed\",\"type\":\"custom\"}"}
{"log.level":"debug","@timestamp":"2023-09-18T14:47:46.433Z","log":{"logger":"elastic-apm-node"},"ctxmgr":"AsyncLocalStorageRunContextManager( RunContext(Transaction(60a6daf2e7e5ed2e, 'unnamed')) )","ecs":{"version":"1.6.0"},"message":"supersedeWithTransRunContext(<Trans 60a6daf2e7e5ed2e>)"}
{"log.level":"debug","@timestamp":"2023-09-18T14:47:46.434Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"setting transaction result {\"trans\":\"60a6daf2e7e5ed2e\",\"trace\":\"1ba8bd8932395bf36c08fd7da8f8ff60\",\"result\":\"HTTP 2xx\"}"}
{"log.level":"debug","@timestamp":"2023-09-18T14:47:46.434Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"setting default transaction name: GET / {\"trans\":\"60a6daf2e7e5ed2e\",\"trace\":\"1ba8bd8932395bf36c08fd7da8f8ff60\"}"}
{"log.level":"debug","@timestamp":"2023-09-18T14:47:46.435Z","log":{"logger":"elastic-apm-node"},"ctxmgr":"AsyncLocalStorageRunContextManager( RunContext() )","ecs":{"version":"1.6.0"},"message":"addEndedTransaction(GET /)"}
{"log.level":"debug","@timestamp":"2023-09-18T14:47:46.435Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"sending transaction {\"trans\":\"60a6daf2e7e5ed2e\",\"trace\":\"1ba8bd8932395bf36c08fd7da8f8ff60\"}"}
{"log.level":"debug","@timestamp":"2023-09-18T14:47:46.435Z","log":{"logger":"elastic-apm-node"},"trans":"60a6daf2e7e5ed2e","name":"GET /","trace":"1ba8bd8932395bf36c08fd7da8f8ff60","type":"request","result":"HTTP 2xx","duration":2.367,"ecs":{"version":"1.6.0"},"message":"ended transaction"}

I have some suspect that NestJs is changing some things from underlying ExpressJs and that causes that transactions cannot read the information correctly. But at the end I couldn't debug and find it by myself. If someone with more experience in Elastic could help with possible reasons and hints would be great.

P.S.
We have some other apps with expressjs and connected to APM without any problem. So Elastic itself is functioning well.