# NPE using APM Agent 1.34.1 **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [October 21, 2022, 9:09am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179 "2022-10-21T09:09:07Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![barath](https://avatars.discourse-cdn.com/v4/letter/b/c0e974/32.png) [@barath](https://discuss.elastic.co/u/barath)\ **Post date:** [October 21, 2022, 9:09am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/1 "2022-10-21T09:09:07Z")
The same error that was reported in [NPE using APM Agent 1.33.0](https://discuss.elastic.co/t/npe-using-apm-agent-1-33-0/312702) still happens in Java agent 1.34.1. ```auto java.lang.NullPointerException at co.elastic.apm.agent.impl.transaction.Span.afterEnd(Span.java:289) at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:601) at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:580) at co.elastic.apm.agent.httpclient.ApacheHttpClientInstrumentation$ApacheHttpClientAdvice.onAfterExecute(ApacheHttpClientInstrumentation.java:99) ``` ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [October 24, 2022, 9:33am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/2 "2022-10-24T09:33:13Z")
Thanks for reporting this @barath , What is the frequency of this issue ? Are you able to reproduce it in your environment ? I have just opened [PR #2859](https://github.com/elastic/apm-agent-java/pull/2859) that should fix this issue. Could you try with [this snapshot](https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-2859/1/artifact/src/github.com/elastic/apm-agent-java/elastic-apm-agent/target/elastic-apm-agent-1.34.2-SNAPSHOT.jar) and tell us if that makes a difference ? ---
**Author:** ![barath](https://avatars.discourse-cdn.com/v4/letter/b/c0e974/32.png) [@barath](https://discuss.elastic.co/u/barath)\ **Post date:** [October 24, 2022, 3:27pm UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/3 "2022-10-24T15:27:58Z")
Thanks for quick response. It happens about three times in an hour (we have about 2300 request per minute). I'll test your fix in our system soon and come back to you when I have results. ---
**Author:** ![barath](https://avatars.discourse-cdn.com/v4/letter/b/c0e974/32.png) [@barath](https://discuss.elastic.co/u/barath)\ **Post date:** [October 26, 2022, 8:56am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/4 "2022-10-26T08:56:26Z")
It seems your fix solved the issue. I deployed the snapshot yesterday afternoon and the error haven't happened since but I've got an another one: ```auto java.lang.NullPointerException at co.elastic.apm.agent.impl.transaction.Span.tryToCompressRegular(Span.java:369) at co.elastic.apm.agent.impl.transaction.Span.tryToCompress(Span.java:317) at co.elastic.apm.agent.impl.transaction.Span.afterEnd(Span.java:286) at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:601) at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:580) at co.elastic.apm.agent.httpclient.ApacheHttpClientInstrumentation$ApacheHttpClientAdvice.onAfterExecute(ApacheHttpClientInstrumentation.java:99) ``` Although it happened only once. ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [October 26, 2022, 9:20am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/5 "2022-10-26T09:20:56Z")
Thanks for checking the snapshot and reporting this other NPE, I'll take a look. Even if it is not triggered that often, it still means something is off here. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [October 31, 2022, 8:33am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/6 "2022-10-31T08:33:09Z")
@barath I attempted to address the recent reported concurrency issue through [PR #2865](https://github.com/elastic/apm-agent-java/pull/2865). Please try [this snapshot](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-java%2Fapm-agent-java-mbp/detail/PR-2865/1/artifacts) to see whether the is resolved. ---
**Author:** ![barath](https://avatars.discourse-cdn.com/v4/letter/b/c0e974/32.png) [@barath](https://discuss.elastic.co/u/barath)\ **Post date:** [November 8, 2022, 9:44am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/7 "2022-11-08T09:44:18Z")
I've deployed your snapshot but unfortunately the exception is still thrown: ```auto java.lang.NullPointerException at co.elastic.apm.agent.impl.transaction.Span.afterEnd(Span.java:289) at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:600) at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:579) at co.elastic.apm.agent.httpclient.ApacheHttpClientInstrumentation$ApacheHttpClientAdvice.onAfterExecute(ApacheHttpClientInstrumentation.java:99) ``` ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [November 13, 2022, 3:06pm UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/8 "2022-11-13T15:06:13Z")
OK, thanks for reporting. I actually recognized there was still a race condition in the former implementation, but didn't get to that so far. I now changed this part of the code to be completely thread-safe, so please try out [this build](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-java%2Fapm-agent-java-mbp/detail/PR-2865/5/artifacts). If there is still an issue, it means I didn't identify the exact issue and will have to look elsewhere. Looking forward to get your feedback. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [November 20, 2022, 1:59pm UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/9 "2022-11-20T13:59:29Z")
@barath did you get the chance to test the latest snapshot? ---
**Author:** ![barath](https://avatars.discourse-cdn.com/v4/letter/b/c0e974/32.png) [@barath](https://discuss.elastic.co/u/barath)\ **Post date:** [November 21, 2022, 1:48pm UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/10 "2022-11-21T13:48:25Z")
@Eyal_Koren Yes, I've deployed the latest snapshot on Tuesday last week. I waited a few days to see what happen. Unfortunately NPE has been thrown twice (on Saturday and today) since Tuesday. ```auto java.lang.NullPointerException at co.elastic.apm.agent.impl.transaction.Span.afterEnd(Span.java:289) at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:604) at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:579) at co.elastic.apm.agent.httpclient.ApacheHttpClientInstrumentation$ApacheHttpClientAdvice.onAfterExecute(ApacheHttpClientInstrumentation.java:99) ``` ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [November 21, 2022, 2:45pm UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/11 "2022-11-21T14:45:09Z")
OK, then I guess it is coming from a different race condition... I'll take a look once I get the chance. Thanks for updating. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [November 22, 2022, 5:36am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/12 "2022-11-22T05:36:31Z")
@barath I had a chance to look at this again this morning. What seems weird is that you get this stack trace consistently. If the latest snapshot is used, this means that [the `Span#parent` field](https://github.com/elastic/apm-agent-java/pull/2865/files#diff-0a29377db9d5c6270ff4865a5585c9cd7633f33eb7a88c57aaa821da9e8c2941R289) is `null`. While I cannot exclude the possibility that some race condition causes this, it should have the same chances of occurring in prior lines, like `Span.java:288`, `Span.java:268` or `Span.java:266`. If you always get it at `Span.java:289`, I would ask you to verify that this is indeed coming from a JVM that indeed uses the latest snapshot, and not some other snapshot/version. Please describe your environment- do you have only one JVM? If you have multiple JVMs, are they logging to the same place? Is there a possibility that this error doesn't come from an agent of the latest snapshot? If not, did you see similar stack traces, but where the top frame in the stack trace is from a different line? ---
**Author:** ![barath](https://avatars.discourse-cdn.com/v4/letter/b/c0e974/32.png) [@barath](https://discuss.elastic.co/u/barath)\ **Post date:** [November 25, 2022, 2:38pm UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/13 "2022-11-25T14:38:25Z")
@Eyal_Koren We have 3 Wildfly instances (Java 11.0.5) running in docker containers on three different hosts. They aren't logging to the same place so stack traces aren't mixed up for sure. The top frame is always the same. Elastic APM agent version is 1.34.2-SNAPSHOT.5079263. Can you confirm this is the latest one? ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [November 29, 2022, 10:57am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/14 "2022-11-29T10:57:52Z")
Yes, this looks like the right snapshot. We'll take another look. ---
**Author:** ![Jonas\_Kunz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonas_kunz/32/110740_2.png) [@Jonas\_Kunz](https://discuss.elastic.co/u/Jonas_Kunz)\ **Post date:** [November 29, 2022, 11:42am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/15 "2022-11-29T11:42:23Z")
Hi @barath, Would it be possible for you to provide us with a full `INFO`-level agent log from the startup until the error occurs in the logs? This might help our investigations. ---
**Author:** ![Jonas\_Kunz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonas_kunz/32/110740_2.png) [@Jonas\_Kunz](https://discuss.elastic.co/u/Jonas_Kunz)\ **Post date:** [November 30, 2022, 11:14am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/16 "2022-11-30T11:14:16Z")
Looks like we found the reason why the error occurs at exactly the line you described. We updated our fix to guard against it. Could you try this [latest snapshot](https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-2865/21/artifact/src/github.com/elastic/apm-agent-java/elastic-apm-agent/target/elastic-apm-agent-1.34.2-SNAPSHOT.jar) which contains the fix and report back whether this fixes the issue for you? Thank you! ---
**Author:** ![barath](https://avatars.discourse-cdn.com/v4/letter/b/c0e974/32.png) [@barath](https://discuss.elastic.co/u/barath)\ **Post date:** [December 1, 2022, 10:37am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/17 "2022-12-01T10:37:52Z")
Hi @Jonas_Kunz , I've deployed the latest snapshot. Let's wait a few days to see what happen. This NPE issue is very rare. ---
**Author:** ![barath](https://avatars.discourse-cdn.com/v4/letter/b/c0e974/32.png) [@barath](https://discuss.elastic.co/u/barath)\ **Post date:** [December 7, 2022, 11:47am UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/18 "2022-12-07T11:47:11Z")
@Jonas_Kunz I haven't got NPE since the update. It can be plain luck or the issue is resolved. Only time can tell. ---
**Author:** ![Jonas\_Kunz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonas_kunz/32/110740_2.png) [@Jonas\_Kunz](https://discuss.elastic.co/u/Jonas_Kunz)\ **Post date:** [December 7, 2022, 12:43pm UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/19 "2022-12-07T12:43:51Z")
Thanks for reporting back! We are confident that we solved the issue. We also added a test which was able to reproduce your specific exception when run on a build without the fix. ---
**Author:** ![barath](https://avatars.discourse-cdn.com/v4/letter/b/c0e974/32.png) [@barath](https://discuss.elastic.co/u/barath)\ **Post date:** [December 7, 2022, 12:55pm UTC](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179/20 "2022-12-07T12:55:55Z")
That's a good news. I'm glad I could help. [Next page](https://discuss.elastic.co/t/npe-using-apm-agent-1-34-1/317179.md?page=2)