# Custom metrics via APM Java agent Micrometer integration are missing **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [October 1, 2020, 10:32pm UTC](https://discuss.elastic.co/t/custom-metrics-via-apm-java-agent-micrometer-integration-are-missing/250746 "2020-10-01T22:32:04Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![muzack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/muzack/32/76548_2.png) [@muzack](https://discuss.elastic.co/u/muzack)\ **Post date:** [October 1, 2020, 10:32pm UTC](https://discuss.elastic.co/t/custom-metrics-via-apm-java-agent-micrometer-integration-are-missing/250746/1 "2020-10-01T22:32:04Z")
Kibana version: 7.9.1 Elasticsearch version: 7.9.1 APM Server version: 7.9.1 APM Agent: Java agent 1.18.0 Java version: 1.8.0\_172 Micrometer version: 1.5.5 Browser version: N/A Original install method: docker-compose recipe Fresh install I'm trying to take the Micrometer automatic integration via the Java APM agent for a spin as described here: [https://www.elastic.co/guide/en/apm/agent/java/current/metrics.html#metrics-micrometer](https://www.elastic.co/guide/en/apm/agent/java/current/metrics.html#metrics-micrometer). Everything looks fine all the way through the v2 intake API, but I see no metrics, nor any errors apparent in the docker console output. In particular I was somewhat expecting a "Metrics" section to appear under my service name in APM, much the same way it does when enabling APM internal instrumentation: Steps to reproduce: 1. Install via docker-compose recipe provided in the APM-\>Deploy-\>Docker documentation. The docker-compose.yaml I'm using is at the end of this description. 2. Configure the agent(just running app via Maven exec), args provided below: ```auto -javaagent:/Users/__/Downloads/elastic-apm-agent-1.18.0.jar -Delastic.apm.service_name=foo -Delastic.apm.log_level=TRACE -Delastic.apm.log_file=${APP_INSTALLDIR}/logs/apm.log -Delastic.apm.server_urls=http://localhost:8200 -Delastic.apm.secret_token= -Delastic.apm.secret_token= -Delastic.apm.application_packages=com.foo ``` Relevant logs: On the app with the agent instrumented things look like they are working: ```auto 2020-10-01 13:59:42,529 [elastic-apm-shared] DEBUG co.elastic.apm.agent.micrometer.MicrometerMetricsReporter - Reporting 2 meters 2020-10-01 13:59:42,532 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving JSON_WRITER event (sequence 7) 2020-10-01 13:59:42,532 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://localhost:8200/intake/v2/events 2020-10-01 13:59:42,535 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10s ``` Logs on APM server from docker console look fine: ```auto apm-server_1 | 2020-10-01T22:17:22.246Z INFO [request] middleware/log_middleware.go:97 request accepted {"request_id": "bc67138c-95ed-4c4b-bae6-d66e7668e5f4", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "pipe", "user-agent": "elasticapm-go/1.8.0 go/go1.13.10", "response_code": 202} ``` But I can find no trace of the metrics in the APM or metricsbeat indices. Here are the indices I see: ```auto yellow open metricbeat-7.9.2-2020.10.01-000001 WdPuUySjRvKek6MtyWGiJw 1 1 44097 0 14.9mb 14.9mb green open .kibana-event-log-7.9.1-000001 Z78jQyAwQrePBj7duE-c5A 1 0 3 0 16.2kb 16.2kb green open .apm-agent-configuration sKfk7YeuTD6Qu2x0UNFl1A 1 0 0 0 208b 208b green open apm-7.9.1-transaction-000001 jiztdQ4GTO-9fweHA_ZnqQ 1 0 13731 0 3.5mb 3.5mb green open apm-7.9.1-profile-000001 Zfblu9xqS2KCudW0L5AqUw 1 0 0 0 208b 208b green open apm-7.9.1-metric-000001 JZEWHg2zQy6ErHBBwlosWg 1 0 43777 0 10.4mb 10.4mb green open .kibana_1 s0uvxRk5SRuIZ3t0BI6QXw 1 0 218 184 10.9mb 10.9mb green open .apm-custom-link SERxmsd0QQKE0B4q61etOw 1 0 0 0 208b 208b green open apm-7.9.1-onboarding-2020.10.01 eYmOai86Tza_gJqpPPNjdA 1 0 1 0 6.8kb 6.8kb green open .kibana_task_manager_1 59BToU3ASM6trf6q_MR3NQ 1 0 6 119 609.8kb 609.8kb green open apm-7.9.1-onboarding-2020.09.28 yGvoFY4dRdm-8iD5q-2OEA 1 0 1 0 6.8kb 6.8kb green open apm-7.9.1-onboarding-2020.09.29 z2KNeJe9R1qvTj2f16E0eQ 1 0 1 0 6.8kb 6.8kb green open apm-7.9.1-span-000001 zJhu2o4mQYWqfFzkvfYw3A 1 0 2139 0 507kb 507kb green open .async-search utsuqrkaTKyYwXXk3VnUZw 1 0 15 2 2.5mb 2.5mb green open apm-7.9.1-error-000001 NFDPNByCRPS9czDLNWJcgw 1 0 91 0 140.7kb 140.7kb ``` docker-compose.yaml: ```auto version: '2.2' services: apm-server: image: docker.elastic.co/apm/apm-server:7.9.1 depends_on: elasticsearch: condition: service_healthy kibana: condition: service_healthy cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"] cap_drop: ["ALL"] ports: - 8200:8200 networks: - elastic command: > apm-server -e -E apm-server.rum.enabled=true -E apm-server.instrumentation.enabled=true -E apm-server.register.ingest.pipeline.enabled=true -E setup.kibana.host=kibana:5601 -E setup.template.settings.index.number_of_replicas=0 -E apm-server.kibana.enabled=true -E apm-server.kibana.host=kibana:5601 -E output.elasticsearch.hosts=["elasticsearch:9200"] healthcheck: interval: 10s retries: 12 test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/ elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1 environment: - bootstrap.memory_lock=true - cluster.name=docker-cluster - cluster.routing.allocation.disk.threshold_enabled=false - discovery.type=single-node - ES_JAVA_OPTS=-XX:UseAVX=2 -Xms1g -Xmx1g ulimits: memlock: hard: -1 soft: -1 volumes: - esdata:/usr/share/elasticsearch/data ports: - 9200:9200 networks: - elastic healthcheck: interval: 20s retries: 10 test: curl -s http://localhost:9200/_cluster/health | grep -vq '"status":"red"' kibana: image: docker.elastic.co/kibana/kibana:7.9.1 depends_on: elasticsearch: condition: service_healthy environment: ELASTICSEARCH_URL: http://elasticsearch:9200 ELASTICSEARCH_HOSTS: http://elasticsearch:9200 ports: - 5601:5601 networks: - elastic healthcheck: interval: 10s retries: 20 test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:5601/api/status volumes: esdata: driver: local networks: elastic: driver: bridge ``` Should there be a `metrics-` index? Anything else I'm missing on config to get custom metrics to appear? Other troubleshooting tips? ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [October 2, 2020, 7:17am UTC](https://discuss.elastic.co/t/custom-metrics-via-apm-java-agent-micrometer-integration-are-missing/250746/2 "2020-10-02T07:17:50Z")
Your micrometer metrics should be in this index: > [@muzack](#): > > `apm-7.9.1-metric-000001` When setting the `log_level` to `trace`, you should also be able to see the sent metrics in the agent logs. ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [October 2, 2020, 2:43pm UTC](https://discuss.elastic.co/t/custom-metrics-via-apm-java-agent-micrometer-integration-are-missing/250746/3 "2020-10-02T14:43:33Z")
Also, please not that if you use an application that has `hikaricp` connection pool, we have a known issue with micrometer integration [https://github.com/elastic/apm/issues/347](https://github.com/elastic/apm/issues/347). If that's the case, using the workaround described in the issue should make it work. If that's the case, you should have warnings in apm-server logs, thus please check that first to make sure it's the issue. ---
**Author:** ![muzack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/muzack/32/76548_2.png) [@muzack](https://discuss.elastic.co/u/muzack)\ **Post date:** [October 2, 2020, 5:55pm UTC](https://discuss.elastic.co/t/custom-metrics-via-apm-java-agent-micrometer-integration-are-missing/250746/4 "2020-10-02T17:55:34Z")
I am seeing the metrics in APM agent logs. Example: ```auto 2020-10-02 10:34:19,869 [elastic-apm-server-reporter] TRACE co.elastic.apm.agent.report.serialize.DslJsonSerializer - {"metricset":{"timestamp":1601660049857000,"samples":{"my.metric.timer.count":{"value":0},"my.metric.timer.sum.us":{"value":0.0},"other.custom.metric.timer.count":{"value":0},"other.custom.metric.timer.sum.us":{"value":0.0}}}} ``` Other metrics(JVM metrics for example) are ending up stored in Elastic just fine in the `apm-7.9.1-metric-000001` index, and `hikaricp` is not in use, so doesn't appear to be related to that issue. Querying for those metrics in that index is turning up nothing: ```auto GET apm-7.9.1-metric/_search { "query": { "exists": { "field": "metricset.samples.my.metric.timer.sum.us" } } } ``` Nor do I see that query matching anything in other indices. In case of replacing `.` with `_` I also checked for field `metricset.samples.my_metric_timer.sum.us` with no matches. And just tried renaming the metrics with underscores, and still don't see any custom metrics stored. Thoughts on next trouble shooting steps? ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [October 5, 2020, 6:13am UTC](https://discuss.elastic.co/t/custom-metrics-via-apm-java-agent-micrometer-integration-are-missing/250746/5 "2020-10-05T06:13:44Z")
Could you try to search indices using wildcards? > [@muzack](#): > > `GET apm-*-metric*/_search` ---
**Author:** ![muzack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/muzack/32/76548_2.png) [@muzack](https://discuss.elastic.co/u/muzack)\ **Post date:** [October 5, 2020, 8:07pm UTC](https://discuss.elastic.co/t/custom-metrics-via-apm-java-agent-micrometer-integration-are-missing/250746/6 "2020-10-05T20:07:40Z")
@felixbarny turns out the query side issue was the top level fields that get indexed are doing some transforms before persisting. Meaning I expected I needed to include `metricset.samples.etc` based on the raw JSON from the agent side logs. If I just search for: ```auto GET apm-7.9.1-metric/_search { "query": { "exists": { "field": "my_timer.count" } } } ``` I get hits, so the data is there. However, I still don't see how to leverage these in the Metrics or APM app. The metrics don't show up in the Metrics Explorer, nor do I see where I'd leverage these from the APM app perspective. Are there some additional steps or doc I'm missing on how to associate my custom metrics with a service? I'll take a closer look at what the APM app internal metrics are doing to see if there are any clues there. Meaning I see an obvious launching point for that service into its metrics, whereas for my app, I see nothing: ![Screen Shot 2020-10-05 at 1.05.31 PM](https://us1.discourse-cdn.com/elastic/original/3X/6/e/6ebffe272df6b1d542e0b7b7bce943757f59881a.png) ![Screen Shot 2020-10-05 at 1.05.58 PM](https://us1.discourse-cdn.com/elastic/original/3X/5/a/5a3b0165f2fed718705106074a7246e11f682b99.png) ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [October 6, 2020, 9:19am UTC](https://discuss.elastic.co/t/custom-metrics-via-apm-java-agent-micrometer-integration-are-missing/250746/7 "2020-10-06T09:19:45Z")
We don't have a good tutorial for that at the moment but I'd recommend to create a Kibana visualization using Lens or TSVB (Time Series Visual Builder). ---
**Author:** ![muzack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/muzack/32/76548_2.png) [@muzack](https://discuss.elastic.co/u/muzack)\ **Post date:** [October 6, 2020, 5:17pm UTC](https://discuss.elastic.co/t/custom-metrics-via-apm-java-agent-micrometer-integration-are-missing/250746/8 "2020-10-06T17:17:23Z")
OK. It's not very cohesive that we can't just create dashboards underneath the APM app and have those dashboards consist of both metrics provided by the APM agent itself, and other custom metrics reported through the agent(like those reported through the Micrometer instrumentation). 😞 But I'll look into those other options for now. Thanks! ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [October 7, 2020, 7:38am UTC](https://discuss.elastic.co/t/custom-metrics-via-apm-java-agent-micrometer-integration-are-missing/250746/9 "2020-10-07T07:38:43Z")
I completely agree with you on that. We're thinking about tighter integrations going forward. But in the meantime, this is hopefully better than having no custom metrics integration at all. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 28, 2020, 3:38am UTC](https://discuss.elastic.co/t/custom-metrics-via-apm-java-agent-micrometer-integration-are-missing/250746/10 "2020-10-28T03:38:53Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.