Java Application Crashed with RejectedExecutionException

Elastic Observability APM
,
1

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version: 7.5

Elasticsearch version: AWS managed

APM Server version: docker.elastic.co/apm/apm-server:7.12.0

APM Agent language and version: Java elastic-apm-agent-1.28.4.jar

Browser version: NA

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Having 8 application which all I want to run with apm-agent but after starting 3 successful services, I start getting below exception on 4th one. We are running all services as an k8 pod.

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@a4dcede rejected from co.elastic.apm.agent.util.ExecutorUtils$1@7ee02968[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3](thread name = elastic-apm-shared)
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:573)
at co.elastic.apm.agent.micrometer.MicrometerMetricsReporter.scheduleReporting(MicrometerMetricsReporter.java:81)
at co.elastic.apm.agent.micrometer.MicrometerMetricsReporter.registerMeterRegistry(MicrometerMetricsReporter.java:67)
at co.elastic.apm.agent.micrometer.MicrometerInstrumentation$AdviceClass.onExit(MicrometerInstrumentation.java:61)
at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:573)
at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:561)
at io.micrometer.core.instrument.MeterRegistry.gauge(MeterRegistry.java:297)
at io.micrometer.core.instrument.Gauge$Builder.register(Gauge.java:190)
at io.micrometer.core.instrument.binder.jvm.JvmGcMetrics.bindTo(JvmGcMetrics.java:109)
at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.lambda$addBinders$1(MeterRegistryConfigurer.java:86)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:390)
at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.addBinders(MeterRegistryConfigurer.java:86)
at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.configure(MeterRegistryConfigurer.java:68)
at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryPostProcessor.postProcessAfterInitialization(MeterRegistryPostProcessor.java:64)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:437)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1790)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1380)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1300)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:541)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213)
at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:212)
at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:203)
at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:97)
at org.springframework.boot.web.servlet.ServletContextInitializerBeans.(ServletContextInitializerBeans.java:86)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:259)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:233)
at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:53)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5219)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:486)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:123)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.(TomcatWebServer.java:104)
at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:450)
at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:199)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:181)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:159)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:577)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:771)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:763)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:438)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:339)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318)
at com.paytmmoney.edtech.transaction.TransactionApplication.main(TransactionApplication.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)

2

Hi @Manoj_Soni , thanks for reporting this.

Does this issue is reproducible and happens every time the application starts or is it a random ? Did it happened more than once since you started using Elastic APM agent ?

Also, could you provide the versions of your dependencies (mvn dependency:tree or gradle equivalent) to know which versions of micrometer/spring boot are involved here ?

From what I understand here the issue is related to the micrometer instrumentation, which for a not-yet known reason is unable to execute an asynchronous task.

As a temporary work-around, you could disable the micrometer instrumentation to prevent this from happening by setting disable_instrumentations=micrometer in your configuration.

3

Hi @Sylvain_Juge , Yes it is reproducible as I have already mentioned that I have 8 micro services after successfully started 3 services with APM agent, when I started 4th services it start throwing this exception. I am sure this is not related to version otherwise it won't start in first three services. I am doubting it's related to CPU resources because not allowing to create new thread or rejected to execute new thread/task. But seems like in appropriate Exception. Is there any limitation on APM server side to configure number of APM agent can connect to server if yes plz do let me know how could I increase that.

Thanks!

4

Hi @Manoj_Soni ,

This issue is not related to apm-server, there is no limitation on the number of apm-agents per apm-server beyond available cpu/memory resources.

Can you confirm if setting disable_instrumentations=micrometer allows to properly start the application ? That would at least allow for a temporary work-around until we investigate the issue.

It is actually a good thing that the issue is reproducible on every start, it helps investigation and allows to validate a fix. The reason we insist on this is because having something that happens randomly and/or due to a timing issue is often way more complicated to diagnose and fix, for example something that fails 10% of the time requires 10x more time and effort to reproduce.

From what I understand you have other similar services that properly start without problem, but for a yet unknown reason this one is slightly different and have this issue.
Thus here, the important is not really knowing what are the exact dependencies versions, but finding the difference between those that work and the one that does not. Beyond dependencies, if you have different JVM or application settings that could help us understand the issue better.

Alternatively, if you start from your application codebase, do you think you could remove/minimize the application code to strict minimal and allow to reproduce the issue ? Having a simple way to reproduce the issue locally on our end would be really awesome.

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.