APM Central configuration not working

Elastic Observability APM
,
1

Kibana version:9.0.3

Elasticsearch version:9.0.3

APM Server version:9.0.3

APM Agent language and version:1.49.0

Browser version: Chrome Version 140.0.7339.128 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: download page

Fresh install or upgraded from other version?

Is there anything special in your setup? I’m running both standalone APM server (on the same host running Elasticsearch and Kibana {WINDOWS} ) on port 8200, and another fleet server (LINUX) with APM integration added on port 8201, both secured with my corp. certificates

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):No matter how many times I try, APM central configuration is not passed on to the agent. UI says “Not yet applied by any agents” and the agent log is pulling empty APM configuration.

image
image1913×517 66.8 KB

image
image1798×290 20.7 KB

Steps to reproduce:

  1. push any transaction from APM agent onto Elasticsearch
  2. disable logging or header captures from APM central configuration
  3. perform the same transaction again and observe (no change)

Errors in browser console (if relevant):none

Provide logs and/or server output (if relevant): APM agent log

2025-09-24 10:37:15,882 [main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2025-09-24 10:37:15,898 [main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific ThreadMXBean interface: com.sun.management.ThreadMXBean
2025-09-24 10:37:15,919 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.49.0 as glassfish-application on Java 17.0.12 Runtime version: 17.0.12+7-LTS VM version: 17.0.12+7-LTS (Red Hat, Inc.) Windows Server 2019 10.0
2025-09-24 10:37:15,920 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - environment: 'payara_95' (source: Java System Properties)
2025-09-24 10:37:15,921 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - capture_body: 'ALL' (source: Java System Properties)
2025-09-24 10:37:15,921 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - trace_methods: 'com.ist.stc.sms.buslogic.generators.*#processReport' (source: Java System Properties)
2025-09-24 10:37:15,921 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - server_url: 'https://10.21.144.84:8201' (source: Java System Properties)
2025-09-24 10:37:15,922 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com' (source: Java System Properties)
2025-09-24 10:37:15,922 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - log_file: 'D:/app-server-payara6/glassfish/domains/domain1/log/KibanaLogs/elastic-apm.log' (source: Java System Properties)
2025-09-24 10:37:17,974 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2025-09-24 10:37:18,740 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2025-06-18T16:03:08Z",  "build_sha": "c60a82e53092e3c579a00205a7af57eeea08144a",  "publish_ready": true,  "version": "9.0.3"}
2025-09-24 10:37:18,771 [elastic-apm-remote-config-poller] WARN  org.stagemonitor.configuration.ConfigurationOption - Error in Java System Properties: Can't convert always to TraceContinuationStrategy Default value 'CONTINUE' for 'trace_continuation_strategy' will be applied.
2025-09-24 10:37:18,771 [elastic-apm-remote-config-poller] INFO  co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Received new configuration from APM Server: {}
2025-09-24 10:37:33,906 [RunLevelControllerThread-1758699443498] INFO  co.elastic.apm.agent.servlet.ServletVersionInstrumentation - Servlet container info = Payara Server 6.2023.12 #badassfish
2025-09-24 10:37:35,177 [RunLevelControllerThread-1758699443498] INFO  co.elastic.apm.agent.servlet.ServletVersionInstrumentation - Servlet container info = Payara Server 6.2023.12 #badassfish
2025-09-24 10:37:36,160 [RunLevelControllerThread-1758699443498] INFO  co.elastic.apm.agent.servlet.ServletVersionInstrumentation - Servlet container info = Payara Server 6.2023.12 #badassfish
2025-09-24 10:37:45,767 [elastic-apm-configuration-reloader] WARN  org.stagemonitor.configuration.ConfigurationOption - Error in Java System Properties: Can't convert always to TraceContinuationStrategy Default value 'CONTINUE' for 'trace_continuation_strategy' will be applied.
2025-09-24 10:38:15,767 [elastic-apm-configuration-reloader] WARN  org.stagemonitor.configuration.ConfigurationOption - Error in Java System Properties: Can't convert always to TraceContinuationStrategy Default value 'CONTINUE' for 'trace_continuation_strategy' will be applied.
2025-09-24 10:38:45,766 [elastic-apm-configuration-reloader] WARN  org.stagemonitor.configuration.ConfigurationOption - Error in Java System Properties: Can't convert always to TraceContinuationStrategy Default value 'CONTINUE' for 'trace_continuation_strategy' will be applied.
2025-09-24 10:39:15,767 [elastic-apm-configuration-reloader] WARN  org.stagemonitor.configuration.ConfigurationOption - Error in Java System Properties: Can't convert always to TraceContinuationStrategy Default value 'CONTINUE' for 'trace_continuation_strategy' will be applied.
2025-09-24 10:39:45,766 [elastic-apm-configuration-reloader] WARN  org.stagemonitor.configuration.ConfigurationOption - Error in Java System Properties: Can't convert always to TraceContinuationStrategy Default value 'CONTINUE' for 'trace_continuation_strategy' will be applied.
2

Hi !

The environment seems properly set, so you should get the expected configuration here.

However, it seems there is an error with trace_continuation_strategy configuration option in your remote configuration (in kibana), as the apparently empty configuration is logged just after the error.

My assumption here is that it might be possible that the invalid value makes the other parts of the remote configuration ignored.

3

Hello @Sylvain_Juge ,

I tried to find anything related to that error but had no luck, is there any simpler configuration to apply just to make sure APM central configuration is working?

4

You should also check that the service name matches the remote configuration and what is configured on the service, if it does not match then it could explain why the configuration remains empty.

5

In the logs, your application name is glassfish-application, in Kibana, for the remote configuration you should make sure the same value is configured for the Service , or that the All value is selected.

In the screenshot you provided, we see that no service is matching this configuration as hinted by “No yet applied by any agents” tooltip on the left of the screenshot.

6

Hello again,

As you can see, the service name and environment are selected from the drop down list in the configuration, i did not type them manually

image
image1611×542 44.7 KB

I have also double checked from the APM traces :

image
image1775×617 47.6 KB

Though I do suspect something else, having 2 APM servers (one standalone and one on fleet server) and both passing traces to the same Elasticsearch/Kibana may have an impact on the agent central configuration.
Right now I’m configuring a new server with only one APM on fleet server (I read somewhere that the agent central configuration only works with APM on fleet) and I will try again.

7

Have you set an explicit service name when adding the java agent to the JVM ?

When not set, then the agent will auto-detect and use one per deployed web-application and report data with distinct values for `service.name`.

However this is not compatible with central configuration. In order to use central configuration you have to setup an explicit service name for the whole JVM. This is a known limitation and can’t be easily solved as having a distinct configuration per application is not possible because some settings are impacting the whole JVM and not just an application.

8

Thanks a lot , that did work!!

Specifying the service name applied the new configuration, now I’m testing whether the central configuration work only on fleet managed agents or all agents.

Just a quick question if you can help, we have multiple servers running glass fish with 5-8 applications each. So when opening APM tab we can see each service (application name) along with their environments (servers)

image
image1660×375 52 KB

When testing the central configuration, I changed the service name on one server to “94_test_service_name” while I was invoking the “Keep Alive Servlet” service.

this means if we change all service names on all servers, this tab would be useless for monitoring, we would see all service names we set manually but wont really know which service is being invoked.

Is there any way to preserve the current view? I saw something at the top called “Service groups” but haven’t read anything about it yet.

9

When you set the service.name explicitly, it means that you’ll have only one visible service per JVM instance, so the Service Inventory view that you showed above would be relevant but only to monitor the overall glassfish instance performance not split per each web-application.

The service groups is a way to group a subset of all the services into a single view, what you’d like to have here is the opposite: splitting one service into multiple sub-services.

Regarding your monitoring though, each application should usually have a dedicated URL path prefix, so you should be able to use the “Traces” view next to the “Service Inventory” to see a high level cross-service overview, but I agree this is not strictly equivalent.

1 Like