Java APM with new SpringBoot 2.6.7 is erroring out on Zuul components

Kim_Attree · May 9, 2022, 12:07pm

Kibana version: 7.16.1

Elasticsearch version: 7.16.1

APM Server version: 7.16.1

APM Agent language and version: 1.30.1

Browser version: n/a

Original install method (e.g. download page, yum, deb, from source, etc.) and version: docker build, java agent installed and added to entrypoint.

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? No

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): n/a

Steps to reproduce:

SpringBoot 2.6.7
Spring Cloud Starter Netflix Zuul 2.2.10.RELEASE
OpenJDK 17 (eclipse-temurin:17.0.3_7-jdk-alpine)
APM Agent 1.30.1

Errors in browser console (if relevant): n/a

java.lang.IllegalStateException: WebApplicationObjectSupport instance [org.springframework.cloud.netflix.zuul.web.ZuulController@7e366c6f] does not run within a ServletContext. Make sure the object is fully configured! 33 
  at org.springframework.web.context.support.WebApplicationObjectSupport.getServletContext(WebApplicationObjectSupport.java:139) 34 
  at org.springframework.web.servlet.mvc.ServletWrappingController.access$200(ServletWrappingController.java:86) 35 
  at org.springframework.web.servlet.mvc.ServletWrappingController$DelegatingServletConfig.getServletContext(ServletWrappingController.java:199) 36 
  at co.elastic.apm.agent.servlet.JavaxUtil.getInfoFromServletContext(JavaxUtil.java:33) 37 
  at co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation$JavaxInit$AdviceClass.onEnter(JavaxServletVersionInstrumentation.java:40) 38 
  at com.netflix.zuul.http.ZuulServlet.init(ZuulServlet.java:55) 39 
  at org.springframework.web.servlet.mvc.ServletWrappingController.afterPropertiesSet(ServletWrappingController.java:153) 40 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1863) 41 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800) 42 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620) 43 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) 44 
  at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) 45 
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) 46 
  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) 47 
  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) 48 
  at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) 49 
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389) 50 
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309) 51 
  at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) 52 
  at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) 53 
  at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:541) 54 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) 55 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) 56 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) 57 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) 58 
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) 59 
  at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) 60  
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) 61 
  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) 62 
  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213) 63 
  at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:270) 64 
  at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:762) 65 
  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:567) 66 
  at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) 67 
  at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:740) 68 
  at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:415) 69 
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) 70 
  at lithium.application.LithiumShutdownSpringApplication.run(LithiumShutdownSpringApplication.java:16) 71 
  at lithium.ui.network.admin.UiNetworkAdminApplication.main(UiNetworkAdminApplication.java:63) 72 
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 73 
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) 74 
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 75 
  at java.base/java.lang.reflect.Method.invoke(Method.java:568) 76 
  at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) 77 
  at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) 78 
  at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) 79 
  at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)

Provide logs and/or server output (if relevant): n/a

Eyal_Koren · May 10, 2022, 7:47am

This error should have no effect on functionality, it only reflects an error trying to discover the Servlet API version for logging purposes. So the fix for this is just catching the exception more quietly. The version may still be discovered later on when a Servlet#service is invoked.
Please try this snapshot and see if the error is eliminated.
Hopefully, everything else works as expected.

Kim_Attree · May 10, 2022, 12:21pm

Hi Eyal,

Using the new snapshot agent jar:

2022-05-10 12:18:21,940 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.30.2-SNAPSHOT.e3f7eee

Still getting that same error for now:

2022-05-10 12:18:42,173 [main] INFO co.elastic.apm.agent.servlet.JavaxUtil - Failed obtain ServletContext from ServletConfig org.springframework.web.servlet.mvc.ServletWrappingController$DelegatingServletConfig@1fdd5517. Stack trace printed in debug level

121

java.lang.IllegalStateException: WebApplicationObjectSupport instance [org.springframework.cloud.netflix.zuul.web.ZuulController@46610fc9] does not run within a ServletContext. Make sure the object is fully configured!

120

at org.springframework.web.context.support.WebApplicationObjectSupport.getServletContext(WebApplicationObjectSupport.java:139)

119

at org.springframework.web.servlet.mvc.ServletWrappingController.access$200(ServletWrappingController.java:86)

118

at org.springframework.web.servlet.mvc.ServletWrappingController$DelegatingServletConfig.getServletContext(ServletWrappingController.java:199)

117

at co.elastic.apm.agent.servlet.adapter.JavaxServletApiAdapter.getServletContextFromServletConfig(JavaxServletApiAdapter.java:223)

116

at co.elastic.apm.agent.servlet.servicename.InitServiceNameInstrumentation$JavaxInitServiceNameInstrumentation$AdviceClass.onEnter(InitServiceNameInstrumentation.java:99)

Eyal_Koren · May 10, 2022, 1:43pm

Hmm, maybe this comes from your Spring error handling configuration...
The agent shouldn't log the stack trace, unless you set it to log at debug level.

Since this should have no functionality effect, maybe the best workaround is to disable this specific instrumentation through the disable_instrumentations config. Since we don't want to disable all servlet instrumentations, I added the ability to disable this one specifically.
Please download this snapshot and try it out with the following configuration: disable_instrumentations=servlet-version. If are not sure how to do that, check out the configuration guide to learn more about available options.

I hope this eliminates the noise and let you enjoy the agent
Looking forward for your feedback.

Kim_Attree · May 10, 2022, 2:51pm

Hi Eyal,

I setup the necessary configuration and confirmed its picked up with the new snapshot:

2022-05-10 14:48:34,087 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.30.2-SNAPSHOT.6b6a317 as ui-network-admin (1.12-82fb5e44) on Java 17.0.3 Runtime version: 17.0.3+7 VM version: 17.0.3+7 (Eclipse Adoptium) Linux 5.4.170+
2022-05-10 14:48:34,088 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - disable_instrumentations: 'servlet-version' (source: Environment Variables)

Doesnt seem to have had the intended effect, same error:

java.lang.IllegalStateException: WebApplicationObjectSupport instance [org.springframework.cloud.netflix.zuul.web.ZuulController@1bd6638] does not run within a ServletContext. Make sure the object is fully configured!
122
at org.springframework.web.context.support.WebApplicationObjectSupport.getServletContext(WebApplicationObjectSupport.java:139)

Eyal_Koren · May 11, 2022, 4:36am

Please share the entire stack trace you get with the latest snapshot and the added configuration.

If possible, set log_level=debug and share the entire log from startup through https://gist.github.com/.
If you set log_file, then it will contain only agent logging.

Kim_Attree · May 11, 2022, 8:12am

Hi @Eyal_Koren I PM'ed you the gist thank you.

Eyal_Koren · May 11, 2022, 10:06am

Thank you for providing it!
It's another instrumentation that we do for Servlet#init (thus suffers from the same too-early getServletContext invocation).
Please try this snapshot, first without the disable_instrumentations config, and if you still get stack traces, or you want to remove the single-line info loggings as well, then try with: disable_instrumentations=servlet-version,servlet-service-name

Eyal_Koren · May 15, 2022, 7:05am

@Kim_Attree I'll be happy to get your feedback based on the instructions above, to know whether we can consider this as handled, or another step required

Eyal_Koren · May 17, 2022, 11:31am

Version 1.31.0 has been released with the related fix. The reported errors should not contain the stack traces above.

system · June 7, 2022, 7:31am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen! java.lang.NoSuchMethodError APM docker , java , agent-explorer	5	695	December 27, 2023
Error in APM agent during startup APM java	3	942	July 28, 2020
Springframework Exception APM	3	693	October 30, 2018
Circular loading of installed providers detected for: Spring Boot >= 3.2.2 APM docker , java	0	317	May 14, 2024
ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen! APM docker , java , server , agent-explorer	12	1233	August 4, 2023

Java APM with new SpringBoot 2.6.7 is erroring out on Zuul components

Related topics