Can't run APM Java agent in Alfresco cluster environment

[Deny7]

Hi,

I'm running Alfresco development environment on two windows 2012 servers in cluster. I was able to configure filebeat, logstash, elastic, kibana for my logs.

Now I want to add APM server and APM Java agent for monitoring.

I did instalation according to elactic documentation, but when I start this server, It gives me this exception:

2019-03-19 09:50:16,848 ERROR [org.springframework.extensions.webscripts.AbstractRuntime] Exception from executeScript: java.io.NotSerializableException:     co.elastic.apm.agent.impl.ContextInScopeRunnableWrapper
com.hazelcast.nio.HazelcastSerializationException: java.io.NotSerializableException: co.elastic.apm.agent.impl.ContextInScopeRunnableWrapper

I can't find this exception on google. I think it's related to that I'm running this server in cluster, because when I instaled agent on my other environment which is not running in cluster, It was working.

I added agent to both nodes of this cluster.

Can anybody help or point me at right direction?

Kibana version:
6.6.1

Elasticsearch version:
6.6.1

APM Server version:
6.6.2

APM Agent language and version:
Java, 1.4.0

[felixbarny (Felix Barnsteiner)]

Hi and thanks a lot for using the Java agent

I have a suspicion what could be the cause of this. Could you paste the full stack trace? That would help me to pinpoint the exact problem.

In the meantime, could you set this configuration:

classes_excluded_from_instrumentation=(?-i)org.infinispan*,(?-i)org.apache.xerces*,(?-i)org.jboss.as.*,(?-i)io.undertow.core*,(?-i)org.eclipse.jdt.ecj*,(?-i)org.wildfly.extension.*,(?-i)org.wildfly.security*,(?-i)com.hazelcast.*

I think the problem is that hazelcast offers Executor implementations which allow the submitted Runnable to be executed on another JVM. But that requires the Runnable to implement the Serializable interface. Because of the instrumentation we do, we wrap the Runnables submitted to an Executor with a non-serializable Runnable.

The config setting will disable instrumentation for hazelcast, so it should work again.

Thanks for reporting this,
Felix

[Deny7]

Thank you very much for quick response. I Added your configuration and It's working. Thanks!!!!!

Stacktrace:

https://justpaste.it/1ugkj

[felixbarny (Felix Barnsteiner)]

I have created a PR to exclude Hazelcast from the default instrumentation: https://github.com/elastic/apm-agent-java/pull/536