[Elasticsearch + APM Server Binary + Distributed Tracing] Latency Distribution: No data to display, and no traces are shown

[shihab.sunny]

Hello everyone,
I have been trying to set up tracing for a java maven spring boot (3.1.2) application using the self managed APM server binary and elastic APM javaagent. Although, I have received necessary trace and span data into my Elasticsearch, but there is no trace graph shown in the Kibana APM UI. Here is what I have tried so far:

Kibana version: 8.6.0

Elasticsearch version: 8.6.0

APM Server version: 8.6.0

APM Agent language and version: Java, 1.52.0 / 1.50.0 (tried with both)

Browser version: Chrome 130.0.6723.117

Fresh install or upgraded from other version?
Fresh install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

• No Logstash or Kafka outputs.
• No load balancer in front of the APM Servers.
• Did not change index pattern
• Did not generate custom templates
• Did not change agent configuration

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

After running the spring boot application with the following command:
java -javaagent:/path/to/elastic-apm-agent-1.52.0.jar -Delastic.apm.service_name=apm-server -Delastic.apm.server_url=http://localhost:8200 -Delastic.apm.environment=my-environment -jar demo.jar

I can see from the console of the spring boot application, that the application has connected to the APM Server (see img: 1 in the embedded image).

Now, if I navigate to Observability -> APM -> Services in Kibana, I can see the service named apm-server. After few requests to the spring boot application, transactions start to show up as well. But when I click a transaction, the latency distribution box is empty. And there is no trace sample as well. Although Throughput and Time spent by span type shows received data (see img: 2, img: 3 in the embedded image).

One error that comes up in the Kibana UI and in the Kibana log upon clicking a transaction can be seen in the embedded image (see img: 4, img: 5 in the embedded image).

Please note that, I have not not set up any agent policy, since it was not mentioned in the below installation guide of APM server binary and javaagent.

Also note that, I have tried the whole thing with 8.12.0 as well. But with no luck

Steps to reproduce:
Everything was installed on the same machine:

1. Install Elasticsearch and Kibana on windows
2. Follow the below guide to install and configure APM server binary
   https://www.elastic.co/guide/en/observability/current/_apm_server_binary.html
3. Follow the below guide to install and configure javaagent
   Manual setup with -javaagent flag | APM Java Agent Reference [1.x] | Elastic

Provide logs and/or server output (if relevant):

Output of /_cat/indices?v is shown in the embedded image (see img: 6 in the embedded image).

** Embedded image**:

[Carson_Ip]

Hi,

Can you confirm if APM integration package is installed? If so, what's the version of it?

I'm asking because it appears in img6 that APM metrics are stored in data streams but not APM traces, which is strange. If APM integration package is installed properly, I'd expect the APM index templates & mappings to work out of the box.

[shihab.sunny]

Hello @Carson_Ip,

Thanks for getting back. You correctly pointed out, the APM integration package was not installed. Since, I was trying to configure everything in an air-gapped environment, upon clicking the Elastic APM page (Observability -> APM -> Add the APM integration), it never got loaded (see img: 1 of the embedded image) and showed error connecting to package registry (see img: 2 of the embedded image). But, I was able to get it to work with an internet connection. However, for an air-gapped environment, I am thinking about following the below thread.

How to Install APM Server in Legacy mode without Elastic APM Integration

I would like to install APM Server in old way so in Legacy. I have few problems with this installation. Could someone explain me two cases: Is it possible to install APM in Legacy mode without this step? Step 2: Set up and configure | APM User Guide [8.6] | Elastic APM Integration is neccessary also in legacy mode? If yes. How it is possible to install APM Integration without access to internet, without any external artifacts, proxies or servers? Is there any binaries or plugins to ins…

Again, Thanks for pointing to the right direction.

Embedded Image:

[Carson_Ip]

Have you enabled air-gapped mode for fleet? See fleet docs. I expect apm integration package page to show up after enabling it. After that you may install the apm integration without any connection to EPR.

Alternatively, you may wait for the 8.15.4 release, and upgrade to it. 8.15 removed the requirement for apm integration package, but 8.15.0 to 8.15.3 have some known issues, and it is advised to use 8.15.4.

[shihab.sunny]

I don't think the property xpack.fleet.isAirGapped: true work with kibana version 8.6.0.

However, I was able to get it to work on Windows by following the above attached thread. Setting up the following property in kibana.yml file did the trick.

xpack.fleet.packages:
  - name: apm
    version: 8.6.0

Note that, it is important to give two spaces before the hyphen sign and one space after the hyphen sign. I could not make it to work otherwise. You need to mention version as well since it is a required field.

Unfortunately, the above solution did not work on Linux. To make it run on Linux, set up the following property in kibana.yml file by following this thread.

xpack.fleet.registryUrl: "http://localhost:8899"

Make sure the port is down i.e. unreachable.


Hopefully, someone finds it helpful.