# Prometheus fields no longer available after enabling APM on Java process **URL:** \ **Category:** APM\ **Tags:** docker, java\ **Created:** [September 27, 2021, 2:56pm UTC](https://discuss.elastic.co/t/prometheus-fields-no-longer-available-after-enabling-apm-on-java-process/285244 "2021-09-27T14:56:15Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![momalley](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@momalley](https://discuss.elastic.co/u/momalley)\ **Post date:** [September 27, 2021, 2:56pm UTC](https://discuss.elastic.co/t/prometheus-fields-no-longer-available-after-enabling-apm-on-java-process/285244/1 "2021-09-27T14:56:15Z")
Prometheus fields no longer available after enabling APM on Java process We are using Kibana 7.10.1 APM Agent 1.24 Browser used Chrome **Steps to reproduce** : 1. We have enabled APM on one of our Java processes -javaagent:/opt/mount1/../global\_resources/elastic-apm-agent-1.24.0.jar -Delastic.apm.service\_name="Name of Service" -Delastic.apm.verify\_server\_cert=false -Delastic.apm.server\_urls=https://{{ apm\_server }}:8200 1. With the above enabled, the APM data metrics get sent to the APM server and is available in Kibana in the APM Dashboard. 2. APM Dashboards show metrics correctly for the Service in question. The problem arises in that some of the prometheus fields/metrics are no longer available to be scraped to Grafana. The prometheus exporter that makes the metrics available to be scraped did not change in any way. Once we disable APM by removing the reference to the jar the prometheus fields/metrics were again available How could it be that the APM jar is updating/changing the metrics made available by the exporter ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [September 28, 2021, 7:43am UTC](https://discuss.elastic.co/t/prometheus-fields-no-longer-available-after-enabling-apm-on-java-process/285244/2 "2021-09-28T07:43:49Z")
Hi @momalley , The agent should not interfere with other ways to capture metrics, if it does then it's definitely an issue we need to investigate and fix. Could you clarify a bit your context here: - how are Prometheus metrics usually collected in your application ? What component/library is in charge of that ? - What are the impacted metrics ? How are they usually captured ? Does it relies on JMX ? - is there any debug/error message that would indicate a different behavior of your prometheus metrics capture process ? ---
**Author:** ![momalley](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@momalley](https://discuss.elastic.co/u/momalley)\ **Post date:** [September 28, 2021, 2:34pm UTC](https://discuss.elastic.co/t/prometheus-fields-no-longer-available-after-enabling-apm-on-java-process/285244/3 "2021-09-28T14:34:44Z")
It does seem strange to us aswell that the APM agent would interfere with how we capture the metrics. We run our java application inside a Docker container. Spring boot is used and the metrics exporter used is [Micrometer Application Monitoring](https://micrometer.io/docs/registry/prometheus) [Spring Metrics](https://docs.spring.io/spring-metrics/docs/current/public/prometheus) MeterRegistry is used for the Metrics import io.micrometer.core.instrument.MeterRegistry The above exporter makes the metrics available on the node so that they can be scraped by Grafana. Some prometheus fields that were available prior to enabling APM are no longer there. We have not seen any errors in the application, just that the prometheus fields are missing. Thanks in advance ---
**Author:** ![momalley](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@momalley](https://discuss.elastic.co/u/momalley)\ **Post date:** [September 28, 2021, 2:37pm UTC](https://discuss.elastic.co/t/prometheus-fields-no-longer-available-after-enabling-apm-on-java-process/285244/4 "2021-09-28T14:37:07Z")
@Sylvain_Juge update above ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [September 29, 2021, 3:58pm UTC](https://discuss.elastic.co/t/prometheus-fields-no-longer-available-after-enabling-apm-on-java-process/285244/5 "2021-09-29T15:58:12Z")
> [@momalley](#): > > Some prometheus fields that were available prior to enabling APM are no longer there Please provide some additional input, so we have a better starting point for analyzing this: - So some are are disappearing and some not? If yes: - Can you find a common factor to the ones not affected vs. the ones affected? For example - [Meter](https://micrometer.io/docs/concepts#_meters) names? Meter types? Meter tags? - Please provide a couple of examples of affected and non-affected meters (metricsets) - What happens if you enable the agent and disable the collection of an affected metric through the [`disable_metrics`](https://www.elastic.co/guide/en/apm/agent/java/current/config-reporter.html#config-disable-metrics) config? Does it restore this metric in Prometheus? - What happens when using the agent but turning off Micrometer instrumentation entirely by setting [`disable_instrumentations=micrometer`](https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-disable-instrumentations)? ---
**Author:** ![momalley](https://avatars.discourse-cdn.com/v4/letter/m/a698b9/32.png) [@momalley](https://discuss.elastic.co/u/momalley)\ **Post date:** [September 30, 2021, 1:21pm UTC](https://discuss.elastic.co/t/prometheus-fields-no-longer-available-after-enabling-apm-on-java-process/285244/6 "2021-09-30T13:21:42Z")
@Eyal_Koren @Sylvain_Juge We have narrowed down the issue a little. We have a scenario with one Java service where the issue does not occur and another scenario where is does occur. For the scenario where the issue happens we see that the service jar file is called using the "-cp" parameters. Below is an example. In particular look at "-cp /etc/hbase/conf:/etc/hadoop/conf:software-ingester.jar" `exec java $(memory_options) -Djavax.net.ssl.trustStore=$java_trust_store -Djava.security.auth.login.config=$kmsw_jaas_path -Dhttps.proxyHost=######## -Dhttps.proxyPort=8080 -Djava.security.egd=file:/dev/./urandom **-cp /etc/hbase/conf:/etc/hadoop/conf:software-ingester.jar** -Dhttps.proxyHost=$https_proxyHost -Dhttps.proxyPort=$https_proxyPort org.springframework.boot.loader.JarLauncher` For another scenario where there are no issues with missing prometheus fields the "-cp" parameters are not used. Should the APM.jar behave any different when the "-cp" parameters are used in the Java call? ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [September 30, 2021, 2:04pm UTC](https://discuss.elastic.co/t/prometheus-fields-no-longer-available-after-enabling-apm-on-java-process/285244/7 "2021-09-30T14:04:14Z")
> Should the APM.jar behave any different when the "-cp" parameters are used in the Java call? It shouldn't, as long as you don't attach the agent jar through `-cp`, which is not allowed. How do you attach the agent? What configurations do you apply for it? Please answer my questions above, so we can properly assist. Providing a `debug` level log (see [`log_level`](https://www.elastic.co/guide/en/apm/agent/java/current/config-logging.html#config-log-level)) may be useful as well, just make sure it includes the entire startup of your application. You can share it through [gist](https://gist.github.com/). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 21, 2021, 10:05am UTC](https://discuss.elastic.co/t/prometheus-fields-no-longer-available-after-enabling-apm-on-java-process/285244/8 "2021-10-21T10:05:12Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [November 23, 2021, 4:02pm UTC](https://discuss.elastic.co/t/prometheus-fields-no-longer-available-after-enabling-apm-on-java-process/285244/9 "2021-11-23T16:02:38Z")
The [fix](https://github.com/elastic/apm-agent-java/pull/2255) for this issue will be soon merged and available in the next version - 1.28.0