# Failed to transform sample model.Sample

**URL:** <https://discuss.elastic.co/t/failed-to-transform-sample-model-sample/249962>\
**Category:** APM\
**Tags:** java, server\
**Created:** [September 25, 2020, 10:44am UTC](https://discuss.elastic.co/t/failed-to-transform-sample-model-sample/249962 "2020-09-25T10:44:29Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![OrangeDog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orangedog/32/4630_2.png) [@OrangeDog](https://discuss.elastic.co/u/OrangeDog)\
**Post date:** [September 25, 2020, 10:44am UTC](https://discuss.elastic.co/t/failed-to-transform-sample-model-sample/249962/1 "2020-09-25T10:44:29Z")

</div>

**Kibana version** : 7.9.2

**Elasticsearch version** : 7.9.2

**APM Server version** : 7.9.2

**APM Agent language and version** : Java 1.18.0

**Browser version** : N/A

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: deb

**Fresh install or upgraded from other version?** upgrade

**Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. Nope

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:  
After a recent upgrade (possibly 7.9.0) apm server logs are full of sample transform errors.

**Steps to reproduce** :

1. Setup APM agent in a server using HikariCP (3.4.5)

**Provide logs and/or server output (if relevant)**:

```
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.idle", Value:2, Values:[]float64(nil), Counts:[]int64(nil)}
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.creation.count", Value:600, Values:[]float64(nil), Counts:[]int64(nil)}
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.creation.sum.us", Value:7.0216e+07, Values:[]float64(nil), Counts:[]int64(nil)}
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.timeout", Value:6, Values:[]float64(nil), Counts:[]int64(nil)}
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.pending", Value:0, Values:[]float64(nil), Counts:[]int64(nil)}
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.creation.sum.us", Value:5.7659e+07, Values:[]float64(nil), Counts:[]int64(nil)}
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.active", Value:0, Values:[]float64(nil), Counts:[]int64(nil)}
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.idle", Value:2, Values:[]float64(nil), Counts:[]int64(nil)}
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.creation.count", Value:511, Values:[]float64(nil), Counts:[]int64(nil)}
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.usage.count", Value:128944, Values:[]float64(nil), Counts:[]int64(nil)}
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.usage.sum.us", Value:1.983578e+09, Values:[]float64(nil), Counts:[]int64(nil)}
Sep 25 10:23:09 apm-server[31357]: WARN [transform] model/metricset.go:141 failed to transform sample model.Sample{Name:"hikaricp.connections.max", Value:50, Values:[]float64(nil), Counts:[]int64(nil)}
```

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [September 28, 2020, 8:29am UTC](https://discuss.elastic.co/t/failed-to-transform-sample-model-sample/249962/2 "2020-09-28T08:29:56Z")

</div>

Hi @OrangeDog,  
are you using micrometer? If so, this might be related to a bug recently found in the Java agent. There is a PR up for fixing it, you can follow along at [apm-agent-java/pull/1419](https://github.com/elastic/apm-agent-java/pull/1419)

---

<div class="post-metadata">

**Author:** ![OrangeDog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orangedog/32/4630_2.png) [@OrangeDog](https://discuss.elastic.co/u/OrangeDog)\
**Post date:** [September 28, 2020, 9:07am UTC](https://discuss.elastic.co/t/failed-to-transform-sample-model-sample/249962/3 "2020-09-28T09:07:18Z")

</div>

Micrometer is there, via Spring Boot.  
This doesn't look like that issue though.

Note the log has errors for both finite float64s and int64s.

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [September 28, 2020, 10:39am UTC](https://discuss.elastic.co/t/failed-to-transform-sample-model-sample/249962/4 "2020-09-28T10:39:21Z")

</div>

Hi @OrangeDog,

- do you have any error/warning on the agent-side ?
- what are the versions of micrometer / spring-boot / other that you are using ?

---

<div class="post-metadata">

**Author:** ![OrangeDog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orangedog/32/4630_2.png) [@OrangeDog](https://discuss.elastic.co/u/OrangeDog)\
**Post date:** [September 28, 2020, 11:53am UTC](https://discuss.elastic.co/t/failed-to-transform-sample-model-sample/249962/5 "2020-09-28T11:53:15Z")

</div>

The only logs I see from the agent look like this:

> 2020-09-28 00:39:52,873 [XNIO-1 task-14] INFO co.elastic.apm.agent.cache.WeakKeySoftValueLoadingCache - The value for the key org.springframework.boot.loader.LaunchedURLClassLoader@5910e440 has been collected, reloading it

It's Spring Boot 2.3.3 (so Micrometer 1.5.4).

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [September 28, 2020, 1:46pm UTC](https://discuss.elastic.co/t/failed-to-transform-sample-model-sample/249962/6 "2020-09-28T13:46:21Z")

</div>

There would be a fast way to help debug this, could you try to set the java agent `log_level=trace` and provide us the lines that contain `metricset` and `hikaricp`?

Something in the lines of

```auto
cat agent.log | grep metricset | grep hikaricp

```

This will make the log output very verbose as it logs everything that the agents send to server.

---

<div class="post-metadata">

**Author:** ![OrangeDog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orangedog/32/4630_2.png) [@OrangeDog](https://discuss.elastic.co/u/OrangeDog)\
**Post date:** [September 28, 2020, 4:37pm UTC](https://discuss.elastic.co/t/failed-to-transform-sample-model-sample/249962/7 "2020-09-28T16:37:21Z")

</div>

Here are some

> Sep 28 16:34:03 env[3405743]: {"metricset":{"timestamp":1601310843073000,"tags":{"pool":"HikariPool-2"},"samples":{"hikaricp.connections.max":{"value":50.0},"hikaricp.connections.usage.count":{"value":46},"[hikaricp.connections.usage.sum.us](http://hikaricp.connections.usage.sum.us)":{"value":409000.0},"hikaricp.connections":{"value":3.0},"hikaricp.connections.active":{"value":0.0},"hikaricp.connections.min":{"value":2.0},"hikaricp.connections.acquire.count":{"value":46},"[hikaricp.connections.acquire.sum.us](http://hikaricp.connections.acquire.sum.us)":{"value":159244.109},"hikaricp.connections.timeout":{"value":0.0},"hikaricp.connections.pending":{"value":0.0},"hikaricp.connections.idle":{"value":3.0},"hikaricp.connections.creation.count":{"value":0},"[hikaricp.connections.creation.sum.us](http://hikaricp.connections.creation.sum.us)":{"value":0.0}}}}  
> Sep 28 16:34:03 env[3405743]: {"metricset":{"timestamp":1601310843073000,"tags":{"pool":"HikariPool-1"},"samples":{"hikaricp.connections.max":{"value":50.0},"hikaricp.connections.usage.count":{"value":50},"[hikaricp.connections.usage.sum.us](http://hikaricp.connections.usage.sum.us)":{"value":889000.0},"hikaricp.connections.acquire.count":{"value":50},"[hikaricp.connections.acquire.sum.us](http://hikaricp.connections.acquire.sum.us)":{"value":217912.23},"hikaricp.connections.min":{"value":2.0},"hikaricp.connections":{"value":3.0},"hikaricp.connections.timeout":{"value":0.0},"hikaricp.connections.pending":{"value":0.0},"hikaricp.connections.active":{"value":0.0},"hikaricp.connections.idle":{"value":3.0},"hikaricp.connections.creation.count":{"value":0},"[hikaricp.connections.creation.sum.us](http://hikaricp.connections.creation.sum.us)":{"value":0.0}}}}  
> Sep 28 16:34:33 env[3405743]: {"metricset":{"timestamp":1601310873073000,"tags":{"pool":"HikariPool-2"},"samples":{"hikaricp.connections.max":{"value":50.0},"hikaricp.connections.usage.count":{"value":60},"[hikaricp.connections.usage.sum.us](http://hikaricp.connections.usage.sum.us)":{"value":416000.0},"hikaricp.connections":{"value":3.0},"hikaricp.connections.active":{"value":0.0},"hikaricp.connections.min":{"value":2.0},"hikaricp.connections.acquire.count":{"value":60},"[hikaricp.connections.acquire.sum.us](http://hikaricp.connections.acquire.sum.us)":{"value":172080.99},"hikaricp.connections.timeout":{"value":0.0},"hikaricp.connections.pending":{"value":0.0},"hikaricp.connections.idle":{"value":3.0},"hikaricp.connections.creation.count":{"value":0},"[hikaricp.connections.creation.sum.us](http://hikaricp.connections.creation.sum.us)":{"value":0.0}}}}  
> Sep 28 16:34:33 env[3405743]: {"metricset":{"timestamp":1601310873073000,"tags":{"pool":"HikariPool-1"},"samples":{"hikaricp.connections.max":{"value":50.0},"hikaricp.connections.usage.count":{"value":64},"[hikaricp.connections.usage.sum.us](http://hikaricp.connections.usage.sum.us)":{"value":896000.0},"hikaricp.connections.acquire.count":{"value":64},"[hikaricp.connections.acquire.sum.us](http://hikaricp.connections.acquire.sum.us)":{"value":243931.7},"hikaricp.connections.min":{"value":2.0},"hikaricp.connections":{"value":3.0},"hikaricp.connections.timeout":{"value":0.0},"hikaricp.connections.pending":{"value":0.0},"hikaricp.connections.active":{"value":0.0},"hikaricp.connections.idle":{"value":3.0},"hikaricp.connections.creation.count":{"value":0},"[hikaricp.connections.creation.sum.us](http://hikaricp.connections.creation.sum.us)":{"value":0.0}}}}  
> Sep 28 16:35:03 env[3405743]: {"metricset":{"timestamp":1601310903073000,"tags":{"pool":"HikariPool-2"},"samples":{"hikaricp.connections.max":{"value":50.0},"hikaricp.connections.usage.count":{"value":74},"[hikaricp.connections.usage.sum.us](http://hikaricp.connections.usage.sum.us)":{"value":423000.0},"hikaricp.connections":{"value":2.0},"hikaricp.connections.active":{"value":0.0},"hikaricp.connections.min":{"value":2.0},"hikaricp.connections.acquire.count":{"value":74},"[hikaricp.connections.acquire.sum.us](http://hikaricp.connections.acquire.sum.us)":{"value":184358.326},"hikaricp.connections.timeout":{"value":0.0},"hikaricp.connections.pending":{"value":0.0},"hikaricp.connections.idle":{"value":2.0},"hikaricp.connections.creation.count":{"value":0},"[hikaricp.connections.creation.sum.us](http://hikaricp.connections.creation.sum.us)":{"value":0.0}}}}  
> Sep 28 16:35:03 env[3405743]: {"metricset":{"timestamp":1601310903073000,"tags":{"pool":"HikariPool-1"},"samples":{"hikaricp.connections.max":{"value":50.0},"hikaricp.connections.usage.count":{"value":78},"[hikaricp.connections.usage.sum.us](http://hikaricp.connections.usage.sum.us)":{"value":903000.0},"hikaricp.connections.acquire.count":{"value":78},"[hikaricp.connections.acquire.sum.us](http://hikaricp.connections.acquire.sum.us)":{"value":269137.9},"hikaricp.connections.min":{"value":2.0},"hikaricp.connections":{"value":2.0},"hikaricp.connections.timeout":{"value":0.0},"hikaricp.connections.pending":{"value":0.0},"hikaricp.connections.active":{"value":0.0},"hikaricp.connections.idle":{"value":2.0},"hikaricp.connections.creation.count":{"value":0},"[hikaricp.connections.creation.sum.us](http://hikaricp.connections.creation.sum.us)":{"value":0.0}}}}

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [September 29, 2020, 2:06pm UTC](https://discuss.elastic.co/t/failed-to-transform-sample-model-sample/249962/8 "2020-09-29T14:06:33Z")

</div>

Thanks to your report, we have identified the source of the issue that is now reported here : [https://github.com/elastic/apm/issues/347](https://github.com/elastic/apm/issues/347)  
The actual issue is due to how apm-server stores metrics, but there is no straightforward way to solve this at the moment.

The impact on application monitoring is that those conflicting metrics are simply ignored.

Currently there is no way to simply ignore those metrics with the [disable\_metrics](https://www.elastic.co/guide/en/apm/agent/java/current/config-reporter.html#config-disable-metrics) configuration option due to [https://github.com/elastic/apm-agent-java/pull/1421](https://github.com/elastic/apm-agent-java/pull/1421) . A fix will be available for this by tomorrow in our [snapshot version](https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=co.elastic.apm&a=elastic-apm-agent&v=LATEST). That will allow to selectively ignore metrics that conflict and keep all the other ones.

In the mean time, the simplest way is to temporary disable micrometer instrumentation by using [disable\_instrumentations](https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-disable-instrumentations). This will disable all micrometer metrics capture, which is a new feature of agent 1.18.0.

`disable_instrumentations=experimental,micrometer`

Or just append `,micrometer` to this comma separated list if a value is already set.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 20, 2020, 10:06am UTC](https://discuss.elastic.co/t/failed-to-transform-sample-model-sample/249962/9 "2020-10-20T10:06:47Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.