When calling a backend API server through a reverse proxy, it is not written to the Elastic APM

Elastic Observability APM
1

Kibana version: 8.14.2

Elasticsearch version: 8.14.2

APM Server version: 8.14.2

APM Agent language and version: .NET 8

Elastic.Apm.NetCoreAll : 1.28.0

nginx server : azure app service
backend api server : azure app service
elastic : azure cloud service

This issue was not present when request the backend API server directly before configuring the nginx server with reverse proxy. Building an nginx server to act as a reverse proxy, it is confirmed that it is not recorded in the Elastic APM.

If you request reverse-proxy-servername.com through the settings below, you will get the result of the backend api normally. If there is a backend api error, it will also be recorded in the logs stream in the elastic.

However, elastic apm services does not record a transaction, which happens only when you request the backend api via reverse-proxy specifically.

server {
    listen       80;
    listen  [::]:80;
    server_name  reverse-proxy-servername;

    #access_log  /var/log/nginx/host.access.log  main;

    location ^~ /api {
        proxy_pass backendapi-server;
        proxy_set_header Host backendapi-server;
        proxy_http_version 1.1;
        proxy_set_header X-Real-IP $remote_addr;
    }

    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
        root /usr/share/nginx/html;
    }
}

As for azure app service, ssl is applied If I request reverse-proxy-servername/api , send it to backendapi-server/api to receive normal results.

CASE 1) request nginx server

  • return ok
  • logging to elastic ok
  • apm to elastic no

CASE 2) request direct backend api server

  • return ok
  • logging to elastic ok
  • apm to elastic ok

CASE 3) request localhost api (with postman) (backend api source)

  • return ok
  • logging to elastic ok
  • apm to elastic ok

Communication between nginx and the backend api server is smooth and results are fine. But why is it not recorded in APM trace only when request backend api through nginx?

When the client calls the backend API directly, it is normally recorded in the elastic app. Therefore, I don't think the .NET source setting itself will be a problem.

additional)
When I request through the nginx server, 'IsSampled' is logged as false and 'dropping unsampled transaction'. When I request the backend API directly, 'IsSampled' collects it normally as true. The backend .NET source does not handle TransactionSampleRateedit, so it is the default (1.0).