We are trying to use the elastic APM Agent with some customizations on it. it works fine with 1.26.2 but when we try to upgrade it to the 1.27.0 or above it starts throwing the above error and the agent doesnt work not the integration tests works.
Below is the stack trace for the same.
> Configure project :
Maven settings.xml file did not contain <sever/> elements : ~/.m2/settings.xml
> Configure project :shared
Maven settings.xml file did not contain <sever/> elements : ~/.m2/settings.xml
> Configure project :sre-agent
Maven settings.xml file did not contain <sever/> elements : ~/.m2/settings.xml
> Configure project :sre-agent-distribution
Maven settings.xml file did not contain <sever/> elements : ~/.m2/settings.xml
> Task :shared:compileKotlin UP-TO-DATE
> Task :shared:compileJava NO-SOURCE
> Task :shared:processResources UP-TO-DATE
> Task :shared:classes UP-TO-DATE
> Task :shared:inspectClassesForKotlinIC UP-TO-DATE
> Task :shared:jar UP-TO-DATE
> Task :compileKotlin UP-TO-DATE
> Task :compileJava NO-SOURCE
> Task :processResources UP-TO-DATE
> Task :classes UP-TO-DATE
> Task :compileTestKotlin
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/azure/storage/datalake/AzureDataTableReaderConfigurationTest.kt: (30, 38): Unnecessary safe call on a non-null receiver of type Message<AzureStorageDataLakeConfiguration>. This expression will have nullable type in future releases
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/azure/storage/datalake/AzureDataTableReaderConfigurationTest.kt: (31, 38): Unnecessary safe call on a non-null receiver of type Message<AzureStorageDataLakeConfiguration>. This expression will have nullable type in future releases
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/azure/storage/datalake/AzureDataTableReaderConfigurationTest.kt: (32, 38): Unnecessary safe call on a non-null receiver of type Message<AzureStorageDataLakeConfiguration>. This expression will have nullable type in future releases
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/azure/storage/datalake/AzureDataTableReaderConfigurationTest.kt: (32, 47): Unnecessary safe call on a non-null receiver of type AzureStorageDataLakeConfiguration. This expression will have nullable type in future releases
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/azure/storage/datalake/AzureDataTableReaderConfigurationTest.kt: (33, 38): Unnecessary safe call on a non-null receiver of type Message<AzureStorageDataLakeConfiguration>. This expression will have nullable type in future releases
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/azure/storage/datalake/AzureDataTableReaderConfigurationTest.kt: (33, 46): Unnecessary safe call on a non-null receiver of type Header. This expression will have nullable type in future releases
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/elasticsearch/ElasticsearchConfigurationReaderTest.kt: (10, 43): 'PreBuiltTransportClient' is deprecated. Deprecated in Java
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/elasticsearch/ElasticsearchConfigurationReaderTest.kt: (26, 13): 'PreBuiltTransportClient' is deprecated. Deprecated in Java
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/jdbc/JdbcConfigurationReaderTest.kt: (42, 101): Unchecked cast: Map<Any!, Any!> to Map<String, String>
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/storageblob/BlobStorageClientConfReaderTest.kt: (104, 16): 'constructor Configuration()' is deprecated. Deprecated in Java
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/storageblob/BlobStorageClientConfReaderTest.kt: (130, 49): Parameter 'expectedProperties' is never used
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/storageblob/BlobStorageClientConfReaderTest.kt: (159, 23): Unnecessary safe call on a non-null receiver of type Iterable<Header>. This expression will have nullable type in future releases
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/storageblob/BlobStorageClientConfReaderTest.kt: (191, 39): Parameter 'value' is never used, could be renamed to _
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/storageblob/BlobStorageClientConfReaderTest.kt: (193, 87): Unnecessary non-null assertion (!!) on a non-null receiver of type String
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/testutils/EnvironmentUtils.kt: (10, 43): Unchecked cast: Any! to MutableMap<String, String>
w: ~/git/sre-agent-plugin/src/test/kotlin/com/walmart/sre/agent/instrumentation/utils/ReflectionUtilsTest.kt: (107, 69): Unchecked cast: Any? to Array<Any>?
> Task :addKotlin
Execution optimizations have been disabled for task ':addKotlin' to ensure correctness due to the following reasons:
- Gradle detected a problem with the following location: '~/git/sre-agent-plugin/build/classes/kotlin/main'. Reason: Task ':compileTestKotlin' uses this output of task ':addKotlin' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency for more details about this problem.
> Task :checkstyleMain NO-SOURCE
> Task :detekt UP-TO-DATE
> Task :shadowJar
> Task :compileTestJava NO-SOURCE
> Task :processTestResources NO-SOURCE
> Task :testClasses UP-TO-DATE
> Task :integration-tests:compileKotlin NO-SOURCE
> Task :integration-tests:compileJava NO-SOURCE
> Task :integration-tests:processResources NO-SOURCE
> Task :integration-tests:classes UP-TO-DATE
> Task :sre-agent:compileKotlin UP-TO-DATE
> Task :sre-agent:compileJava NO-SOURCE
> Task :sre-agent:processResources UP-TO-DATE
> Task :sre-agent:classes UP-TO-DATE
> Task :sre-agent:inspectClassesForKotlinIC
> Task :sre-agent:shadowJar UP-TO-DATE
> Task :sre-agent:jar
> Task :integration-tests:compileTestKotlin
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/cosmos/AbstractCosmosTest.kt: (117, 41): Unsafe use of a nullable receiver of type CosmosDatabaseResponse?
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/elastic/ElasticApmTracerInstrumentationTest.kt: (19, 8): Unable to find overload of function kotlin.test.assertTrue for power-assert transformation callable as:
- kotlin.test.assertTrue(kotlin.String?, String)
- kotlin.test.assertTrue(kotlin.String?, () -> String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, () -> String)
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/elastic/SpanInstrumentationTest.kt: (15, 25): Type mismatch: inferred type is ElasticApmTracer? but ElasticApmTracer was expected
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/elastic/SpanInstrumentationTest.kt: (23, 8): Unable to find overload of function kotlin.test.assertTrue for power-assert transformation callable as:
- kotlin.test.assertTrue(kotlin.String?, String)
- kotlin.test.assertTrue(kotlin.String?, () -> String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, () -> String)
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/elastic/SpanInstrumentationTest.kt: (24, 8): Unable to find overload of function kotlin.test.assertTrue for power-assert transformation callable as:
- kotlin.test.assertTrue(kotlin.String?, String)
- kotlin.test.assertTrue(kotlin.String?, () -> String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, () -> String)
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/elastic/SpanInstrumentationTest.kt: (25, 8): Unable to find overload of function kotlin.test.assertTrue for power-assert transformation callable as:
- kotlin.test.assertTrue(kotlin.String?, String)
- kotlin.test.assertTrue(kotlin.String?, () -> String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, () -> String)
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/elastic/TransactionInstrumentationTest.kt: (15, 39): Type mismatch: inferred type is ElasticApmTracer? but ElasticApmTracer was expected
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/elastic/TransactionInstrumentationTest.kt: (23, 8): Unable to find overload of function kotlin.test.assertTrue for power-assert transformation callable as:
- kotlin.test.assertTrue(kotlin.String?, String)
- kotlin.test.assertTrue(kotlin.String?, () -> String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, () -> String)
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/elastic/TransactionInstrumentationTest.kt: (24, 8): Unable to find overload of function kotlin.test.assertTrue for power-assert transformation callable as:
- kotlin.test.assertTrue(kotlin.String?, String)
- kotlin.test.assertTrue(kotlin.String?, () -> String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, () -> String)
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/elastic/TransactionInstrumentationTest.kt: (25, 8): Unable to find overload of function kotlin.test.assertTrue for power-assert transformation callable as:
- kotlin.test.assertTrue(kotlin.String?, String)
- kotlin.test.assertTrue(kotlin.String?, () -> String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, String)
- kotlin.test.assertTrue(kotlin.String?, kotlin.Function0<kotlin.Boolean>, () -> String)
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/elasticsearch/ElasticsearchInstrumentationTest.kt: (12, 43): 'PreBuiltTransportClient' is deprecated. Deprecated in Java
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/elasticsearch/ElasticsearchInstrumentationTest.kt: (32, 9): 'PreBuiltTransportClient' is deprecated. Deprecated in Java
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/mongo/MongoContainerTest.kt: (67, 49): 'count(): Long' is deprecated. Deprecated in Java
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/resilience4j/CircuitbreakerConfigTests.kt: (33, 32): Parameter 'e' is never used, could be renamed to _
w: ~/git/sre-agent-plugin/integration-tests/src/test/kotlin/com/walmart/sre/agent/instrumentation/resilience4j/CircuitbreakerConfigTests.kt: (44, 16): Condition 'cb != null' is always 'true'
> Task :integration-tests:compileTestJava NO-SOURCE
> Task :integration-tests:processTestResources
> Task :integration-tests:testClasses
> Task :integration-tests:test
2023-01-06 17:15:02,753 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2023-01-06 17:15:02,805 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM unknown as GradleWorkerMain (null) on Java 1.8.0_292 Runtime version: 1.8.0_292-b10 VM version: 25.292-b10 (AdoptOpenJDK) Mac OS X 10.16
2023-01-06 17:15:02,805 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - plugins_dir: '~/git/sre-agent-plugin/build/libs' (source: Java System Properties)
2023-01-06 17:15:02,805 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - disable_send: 'true' (source: Java System Properties)
2023-01-06 17:15:02,806 [main] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and decrease startup time, please configure application_packages
2023-01-06 17:15:02,899 [main] INFO co.elastic.apm.agent.bci.ElasticApmAgent - Loading plugin sre-agent-plugin-0.0.0-SNAPSHOT.jar
2023-01-06 17:15:05,913 [elastic-apm-metadata-1] WARN co.elastic.apm.agent.impl.metadata.CloudMetadataProvider - Unexpected error during automatic discovery process for cloud provider
java.util.concurrent.TimeoutException: null
at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_292]
at co.elastic.apm.agent.impl.metadata.CloudMetadataProvider.tryAllCloudProviders(CloudMetadataProvider.java:177) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at co.elastic.apm.agent.impl.metadata.CloudMetadataProvider.fetchAndParseCloudProviderInfo(CloudMetadataProvider.java:119) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at co.elastic.apm.agent.impl.metadata.CloudMetadataProvider.getCloudInfoProvider(CloudMetadataProvider.java:71) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at co.elastic.apm.agent.impl.metadata.MetaData$2.call(MetaData.java:109) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at co.elastic.apm.agent.impl.metadata.MetaData$2.call(MetaData.java:105) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
2023-01-06 17:15:21,051 [main] WARN co.elastic.apm.agent.bci.bytebuddy.ErrorLoggingListener - Error on transformation com.lmax.disruptor.dsl.BasicExecutor
java.lang.IllegalArgumentException: No advice defined by class co.elastic.apm.agent.concurrent.ExecutorInstrumentation$ExecutorRunnableInstrumentation$AdviceClass
at net.bytebuddy.asm.Advice.to(Advice.java:351) ~[sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.asm.Advice$WithCustomMapping.to(Advice.java:14250) ~[sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$Transformer$ForAdvice$Entry$ForUnifiedAdvice.resolve(AgentBuilder.java:3056) ~[sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$Transformer$ForAdvice.transform(AgentBuilder.java:2842) ~[sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:11739) ~[sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11677) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1700(AgentBuilder.java:11395) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12069) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12016) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292]
at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11584) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at sun.instrument.TransformerManager.transform(TransformerManager.java:188) [?:1.8.0_292]
at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428) ~[?:1.8.0_292]
at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method) [?:1.8.0_292]
at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144) [?:1.8.0_292]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_292]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
at net.bytebuddy.utility.Invoker$Dispatcher.invoke(Unknown Source) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.utility.dispatcher.JavaDispatcher$Dispatcher$ForNonStaticMethod.invoke(JavaDispatcher.java:1013) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.utility.dispatcher.JavaDispatcher$ProxiedInvocationHandler.invoke(JavaDispatcher.java:1142) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.$Proxy26.retransformClasses(Unknown Source) [?:?]
at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(AgentBuilder.java:7979) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(AgentBuilder.java:7794) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy.apply(AgentBuilder.java:5616) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:10779) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:10687) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at net.bytebuddy.agent.builder.AgentBuilder$Default$Delegator.installOn(AgentBuilder.java:12417) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:269) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:160) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at co.elastic.apm.agent.bci.ElasticApmAgent.initialize(ElasticApmAgent.java:146) [sre-agent-0.0.0-SNAPSHOT.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_292]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
at co.elastic.apm.agent.premain.AgentMain.loadAndInitializeAgent(AgentMain.java:155) [sre-agent-0.0.0-SNAPSHOT.jar:1.27.0]
at co.elastic.apm.agent.premain.AgentMain.init(AgentMain.java:93) [sre-agent-0.0.0-SNAPSHOT.jar:1.27.0]
at co.elastic.apm.agent.premain.AgentMain.premain(AgentMain.java:49) [sre-agent-0.0.0-SNAPSHOT.jar:1.27.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_292]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) [?:1.8.0_292]
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) [?:1.8.0_292]