Java APM with new SpringBoot 2.6.7 is erroring out on Zuul components

Kibana version: 7.16.1

Elasticsearch version: 7.16.1

APM Server version: 7.16.1

APM Agent language and version: 1.30.1

Browser version: n/a

Original install method (e.g. download page, yum, deb, from source, etc.) and version: docker build, java agent installed and added to entrypoint.

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? No

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): n/a

Steps to reproduce:

  1. SpringBoot 2.6.7
  2. Spring Cloud Starter Netflix Zuul 2.2.10.RELEASE
  3. OpenJDK 17 (eclipse-temurin:17.0.3_7-jdk-alpine)
  4. APM Agent 1.30.1

Errors in browser console (if relevant): n/a

java.lang.IllegalStateException: WebApplicationObjectSupport instance [org.springframework.cloud.netflix.zuul.web.ZuulController@7e366c6f] does not run within a ServletContext. Make sure the object is fully configured! 33 
  at org.springframework.web.context.support.WebApplicationObjectSupport.getServletContext(WebApplicationObjectSupport.java:139) 34 
  at org.springframework.web.servlet.mvc.ServletWrappingController.access$200(ServletWrappingController.java:86) 35 
  at org.springframework.web.servlet.mvc.ServletWrappingController$DelegatingServletConfig.getServletContext(ServletWrappingController.java:199) 36 
  at co.elastic.apm.agent.servlet.JavaxUtil.getInfoFromServletContext(JavaxUtil.java:33) 37 
  at co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation$JavaxInit$AdviceClass.onEnter(JavaxServletVersionInstrumentation.java:40) 38 
  at com.netflix.zuul.http.ZuulServlet.init(ZuulServlet.java:55) 39 
  at org.springframework.web.servlet.mvc.ServletWrappingController.afterPropertiesSet(ServletWrappingController.java:153) 40 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1863) 41 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800) 42 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620) 43 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) 44 
  at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) 45 
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) 46 
  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) 47 
  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) 48 
  at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) 49 
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389) 50 
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309) 51 
  at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) 52 
  at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) 53 
  at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:541) 54 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) 55 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) 56 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) 57 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) 58 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) 59 
  at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) 60  
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) 61 
  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) 62 
  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213) 63 
  at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:270) 64 
  at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:762) 65 
  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:567) 66 
  at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) 67 
  at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:740) 68 
  at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:415) 69 
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) 70 
  at lithium.application.LithiumShutdownSpringApplication.run(LithiumShutdownSpringApplication.java:16) 71 
  at lithium.ui.network.admin.UiNetworkAdminApplication.main(UiNetworkAdminApplication.java:63) 72 
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 73 
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) 74 
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 75 
  at java.base/java.lang.reflect.Method.invoke(Method.java:568) 76 
  at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) 77 
  at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) 78 
  at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) 79 
  at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)

Provide logs and/or server output (if relevant): n/a

This error should have no effect on functionality, it only reflects an error trying to discover the Servlet API version for logging purposes. So the fix for this is just catching the exception more quietly. The version may still be discovered later on when a Servlet#service is invoked.
Please try this snapshot and see if the error is eliminated.
Hopefully, everything else works as expected.

Hi Eyal,

Using the new snapshot agent jar:

2022-05-10 12:18:21,940 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.30.2-SNAPSHOT.e3f7eee

Still getting that same error for now:

2022-05-10 12:18:42,173 [main] INFO co.elastic.apm.agent.servlet.JavaxUtil - Failed obtain ServletContext from ServletConfig org.springframework.web.servlet.mvc.ServletWrappingController$DelegatingServletConfig@1fdd5517. Stack trace printed in debug level

121

java.lang.IllegalStateException: WebApplicationObjectSupport instance [org.springframework.cloud.netflix.zuul.web.ZuulController@46610fc9] does not run within a ServletContext. Make sure the object is fully configured!

120

at org.springframework.web.context.support.WebApplicationObjectSupport.getServletContext(WebApplicationObjectSupport.java:139)

119

at org.springframework.web.servlet.mvc.ServletWrappingController.access$200(ServletWrappingController.java:86)

118

at org.springframework.web.servlet.mvc.ServletWrappingController$DelegatingServletConfig.getServletContext(ServletWrappingController.java:199)

117

at co.elastic.apm.agent.servlet.adapter.JavaxServletApiAdapter.getServletContextFromServletConfig(JavaxServletApiAdapter.java:223)

116

at co.elastic.apm.agent.servlet.servicename.InitServiceNameInstrumentation$JavaxInitServiceNameInstrumentation$AdviceClass.onEnter(InitServiceNameInstrumentation.java:99)

Hmm, maybe this comes from your Spring error handling configuration...
The agent shouldn't log the stack trace, unless you set it to log at debug level.

Since this should have no functionality effect, maybe the best workaround is to disable this specific instrumentation through the disable_instrumentations config. Since we don't want to disable all servlet instrumentations, I added the ability to disable this one specifically.
Please download this snapshot and try it out with the following configuration: disable_instrumentations=servlet-version. If are not sure how to do that, check out the configuration guide to learn more about available options.

I hope this eliminates the noise and let you enjoy the agent :slight_smile:
Looking forward for your feedback.

Hi Eyal,

I setup the necessary configuration and confirmed its picked up with the new snapshot:

2022-05-10 14:48:34,087 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.30.2-SNAPSHOT.6b6a317 as ui-network-admin (1.12-82fb5e44) on Java 17.0.3 Runtime version: 17.0.3+7 VM version: 17.0.3+7 (Eclipse Adoptium) Linux 5.4.170+
2022-05-10 14:48:34,088 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - disable_instrumentations: 'servlet-version' (source: Environment Variables)

Doesnt seem to have had the intended effect, same error:

java.lang.IllegalStateException: WebApplicationObjectSupport instance [org.springframework.cloud.netflix.zuul.web.ZuulController@1bd6638] does not run within a ServletContext. Make sure the object is fully configured!
122
at org.springframework.web.context.support.WebApplicationObjectSupport.getServletContext(WebApplicationObjectSupport.java:139)

Please share the entire stack trace you get with the latest snapshot and the added configuration.

If possible, set log_level=debug and share the entire log from startup through https://gist.github.com/.
If you set log_file, then it will contain only agent logging.

Hi @Eyal_Koren I PM'ed you the gist thank you.

Thank you for providing it!
It's another instrumentation that we do for Servlet#init (thus suffers from the same too-early getServletContext invocation).
Please try this snapshot, first without the disable_instrumentations config, and if you still get stack traces, or you want to remove the single-line info loggings as well, then try with: disable_instrumentations=servlet-version,servlet-service-name

@Kim_Attree I'll be happy to get your feedback based on the instructions above, to know whether we can consider this as handled, or another step required

Version 1.31.0 has been released with the related fix. The reported errors should not contain the stack traces above.