APM FileSystemAlreadyExistsException

Elastic Observability APM
1

APM Server version: 8.18.2

APM Agent language and version: Java 1.55.0

Is there anything special in your setup? This is a Scala application running on Kubernetes with embedded Jetty server. Entrypoint for the container is:

#!/bin/sh

echo "$(java -version)"
exec java -server \
          -javaagent:/opt/login-api/elastic-apm-agent-1.55.0.jar \
          -javaagent:/opt/login-api/jmx_prometheus_javaagent-1.0.1.jar=8081:/opt/login-api/jmx_prometheus_config.yaml \
          --add-opens=java.base/java.lang=ALL-UNNAMED \
          -Denvironment="$ENVIRONMENT" \
          -Djava.io.tmpdir=/var/tmp \
          -Dfile.encoding=UTF-8 \
          -Dconfig.resource="$ENVIRONMENT".conf \
          -Dlog.local.threshold=OFF \
          -Dlog.kibana.threshold="$LOG_LEVEL" \
          -Dlog.container_id="$HOSTNAME" \
          -Delastic.apm.service_name=login-api-"$ENVIRONMENT" \
          -Delastic.apm.application_packages=my.package \
          -Delastic.apm.use_path_as_transaction_name=true \
          -jar /opt/login-api/login-api.jar

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): I’m getting a FileSystemAlreadyExistsException when starting the Java APM agent in my Scala app. The problem appeared after I changed the initialization from the ElasticApmAttacher to loading the agent manually with the -javaagent flag.

Provide logs and/or server output (if relevant):

2025-08-26 07:31:32,614 [main] ERROR co.elastic.apm.agent.bci.IndyBootstrap - null
java.nio.file.FileSystemAlreadyExistsException: null
     at jdk.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:104) ~[jdk.zipfs:?]
     at java.nio.file.FileSystems.newFileSystem(FileSystems.java:339) ~[?:?]
     at java.nio.file.FileSystems.newFileSystem(FileSystems.java:288) ~[?:?]
     at co.elastic.apm.agent.util.PackageScanner.doGetClassNames(PackageScanner.java:91) ~[login-api.jar:1.0.0]
     at co.elastic.apm.agent.util.PackageScanner.getClassNames(PackageScanner.java:61) ~[login-api.jar:1.0.0]
     at co.elastic.apm.agent.bci.IndyBootstrap.getClassNamesFromBundledPlugin(IndyBootstrap.java:549) ~[login-api.jar:1.0.0]
     at co.elastic.apm.agent.bci.IndyBootstrap.internalBootstrap(IndyBootstrap.java:476) ~[login-api.jar:1.0.0]
     at co.elastic.apm.agent.bci.IndyBootstrap.bootstrap(IndyBootstrap.java:416) ~[login-api.jar:1.0.0]
     at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
     at java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
     at java.lang.IndyBootstrapDispatcher.bootstrap(IndyBootstrapDispatcher.java:61) ~[?:?]
     at java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:147) ~[?:?]
     at java.lang.invoke.CallSite.makeSite(CallSite.java:316) ~[?:?]
     at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:274) ~[?:?]
     at java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:264) ~[?:?]
     at org.scalatra.servlet.ScalatraListener.contextInitialized(ScalatraListener.scala:21) ~[login-api.jar:1.0.0]
     at org.eclipse.jetty.ee10.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:1589) ~[login-api.jar:1.0.0]
     at org.eclipse.jetty.ee10.servlet.ServletContextHandler.contextInitialized(ServletContextHandler.java:500) ~[login-api.jar:1.0.0]
     at org.eclipse.jetty.ee10.servlet.ServletHandler.initialize(ServletHandler.java:675) ~[login-api.jar:1.0.0]
     at org.eclipse.jetty.ee10.servlet.ServletContextHandler.startContext(ServletContextHandler.java:1323) ~[login-api.jar:1.0.0]
     at org.eclipse.jetty.ee10.webapp.WebAppContext.startWebapp(WebAppContext.java:1350) ~[login-api.jar:1.0.0]
     at org.eclipse.jetty.ee10.webapp.WebAppContext.startContext(WebAppContext.java:1308) ~[login-api.jar:1.0.0]
     at org.eclipse.jetty.ee10.servlet.ServletContextHandler.lambda$doStart$0(ServletContextHandler.java:1049) ~[login-api.jar:1.0.0]
     at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.call(ContextHandler.java:1456) [login-api.jar:1.0.0]
     at org.eclipse.jetty.ee10.servlet.ServletContextHandler.doStart(ServletContextHandler.java:1046) [login-api.jar:1.0.0]
     at org.eclipse.jetty.ee10.webapp.WebAppContext.doStart(WebAppContext.java:504) [login-api.jar:1.0.0]
     at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) [login-api.jar:1.0.0]
     at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [login-api.jar:1.0.0]
     at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:120) [login-api.jar:1.0.0]
     at org.eclipse.jetty.server.Handler$Abstract.doStart(Handler.java:491) [login-api.jar:1.0.0]
     at org.eclipse.jetty.server.Handler$Abstract.doStart(Handler.java:491) [login-api.jar:1.0.0]
     at org.eclipse.jetty.server.handler.StatisticsHandler.doStart(StatisticsHandler.java:66) [login-api.jar:1.0.0]
     at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) [login-api.jar:1.0.0]
     at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [login-api.jar:1.0.0]
     at org.eclipse.jetty.server.Server.start(Server.java:641) [login-api.jar:1.0.0]
     at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:120) [login-api.jar:1.0.0]
     at org.eclipse.jetty.server.Handler$Abstract.doStart(Handler.java:491) [login-api.jar:1.0.0]
     at org.eclipse.jetty.server.Server.doStart(Server.java:582) [login-api.jar:1.0.0]
     at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) [login-api.jar:1.0.0]
     at my.package.Jetty.startInternal(Jetty.scala:44) [login-api.jar:1.0.0]
     at my.package.Jetty.<init>(Jetty.scala:14) [login-api.jar:1.0.0]
     at my.package.jettyProduction$.main(jettyProduction.scala:15) [login-api.jar:1.0.0]
     at my.package.jettyProduction.main(jettyProduction.scala) [login-api.jar:1.0.0]
2

Hi !

Thanks for reporting this issue.

Initially you used the programatic setup and you switched to the -javaagent setup, is that correct ?

Did you removed the ElasticApmAttacher and the dependencies to the apm-agent-attach jar from the application ?

Also, can you reproduce the issue locally outside the deployment environment ?

If the answer is YES to both of the questions above, could you provide a minimal application that allows to reproduce it so we can investigate ? It does not need to run in docker/k8s, the problem is more likely related to the java application itself.

3

Yes. I switched to the -javaagent because the dynamic agent loading will be disabled in a future Java release and I wanted to future proof our application.

Yes, I did.

No, I haven’t been able to reproduce this locally yet. Running the application through SBT seems to work just fine. I’ll see if I can reproduce it and if I can make a minimal application where this happen.

4

I managed to reproduce the error on my machine but haven’t yet been able to create a minimal example application to share with you, unfortunately.

5

I think I’ve managed to create a minimal reproduction of this issue: GitHub - tophattom/my-scalatra-app

Building a JAR and running it using the commands in the README should produce the exception in question. It is probably the most minimal example but hopefully still helpful.

6

@Sylvain_Juge have you had the chance to try out the example app?

7

Hi @jrf , I’ve managed to reproduce the issue thanks to your provided application, unfortunately this will probably take a bit of work to fix it as it seems to be class-loading related.

As an alternative to the Elastic APM Java agent, you could use the EDOT Java, which provide similar features, but relies on OpenTelemetry instrumentation. I’ve tried it on your provided app and it does seem to work as expected, at least I don’t get this exception.

8

Well, after a bit of investigation I’ve found that the issue is actually a Jetty bug and I have filed an issue on the jetty project: unexpected FileSystemAlreadyExistsException related to ResourceFactoryInternals.mountIfNeeded · Issue #13532 · jetty/jetty.project · GitHub, I’ve used your test application as an easy way to reproduce, so if you can keep it around for a bit it might help solve it on the Jetty side.

The exception is also thrown when the apm agent is not attached, it just happens to be silently ignored by jetty internal implementation. When the agent indirectly triggers it through classloading and instrumentation it reports it as an error, but unlike the jetty implementation it can’t ignore the unexpected exception and then reports it in the application standard output.

The good news however is that this exception is likely not impacting the normal application behavior, the fact that the agent makes it verbose and noisy is unfortunate though.

9

Thanks for looking into this! :folded_hands: We are not in a hurry with this so I’ll wait a bit and see if the Jetty issue gets (or is already) solved. :slight_smile:

10

The initial issue is fixed by updating to latest jetty version 12.1.0, however a very similar bug is now being triggered by the Elastic APM Java agent, so I’ve opened fix FileSystemAlreadyExistsException with package scanner by SylvainJuge · Pull Request #4204 · elastic/apm-agent-java · GitHub to fix it, this will likely be included in the next release.

1 Like