Elastic APM service discovery not working with public API

Elastic Observability APM
1

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version: v8.2.0 (Elastic Cloud)

Elasticsearch version:

APM Server version: v8.2.0 (Elastic Cloud)

APM Agent language and version: Java 1.31.0

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Elastic Cloud

Fresh install or upgraded from other versions?: Fresh

Is there anything special in your setup? No, services communicate through Public APIs

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

The inter service communication between services in a microservice based application is not accurate for the services communicating with each other through public APIs. Elastic APM considers the public endpoints as a different service and does not actually connect with the service that is instrumented with Elastic APM. The services are hosted on the same cluster of Azure Spring Cloud and all services use Java Spring with the same elastic agent version.

The actual flow of transaction -

image
image1491脳840 43.5 KB

Elastic APM Service Map-
image

As you can see, there are no links to produce service but there are new services for the public endpoints called

Steps to reproduce:

  1. Deploy a simple microservice based application that communicates with each other through publically exposed REST APIs.
  2. Instrument one service with APM and perform a transaction that hits all services in the application.
  3. Instrument the rest of the application for all services.
  4. View the service map in the kibana dashboard and the service links are not accurate

Errors in browser console (if relevant): N/A

Provide logs and/or server output (if relevant):

2

Hi @avinashupadhyaya

Do you see any transactions in produce-service?
What http client do you use to communicate between Order Service and Produce service services?

3

Hi @nugusbayevkk
There are no transactions in the produce service.
I am using the rest template in Java for the HTTP calls. Code here - order-service/ProduceService.java at f2dd2657a150fd60d39587587bf1f1146f90cf21 路 avinashupadhya99/order-service 路 GitHub

4

I analyzed your services, all services are supported by agent.
Can you provide information how do you run elastic-java-agent with your Order-Service and Produce Service?

5

Thank you for validating the services. The services are instrumented using the -javaagent flag while deploying the services to Azure Spring Cloud. I have followed How to monitor Spring Boot apps with Elastic APM Java Agent | Microsoft Docs

6

@avinashupadhyaya

If possible, set log_level=debug (Logging configuration options | APM Java Agent Reference [master] | Elastic) in Produce Service and share the entire log from startup through https://gist.github.com/.

In addition can you provide detailed information how do you deploy your applications on azure for Produce Service and Order Service?

az spring app deploy \
    --name <your-app-name> \
    --artifact-path <unique-path-to-your-app-jar-on-custom-storage> \
    --jvm-options='-javaagent:<agent-location>' \
    --env ELASTIC_APM_SERVICE_NAME=<your-app-name> \
          ELASTIC_APM_APPLICATION_PACKAGES='<your-app-package-name>' \
          ELASTIC_APM_SERVER_URL='<your-Elastic-APM-server-URL>' \
          ELASTIC_APM_SECRET_TOKEN='hide value'
7

I have documented the deployment steps on azure-spring-cloud-apm.md 路 GitHub

When I turn on the debug logs, the following were captured for produce-service - elastic-apm-debug-logs.log 路 GitHub

8

Hi @avinashupadhyaya ,
can you provide debug logs from application startup?

9

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.