Apm-agent not working, after updating war in tomcat

Elastic Observability APM
1

Kibana version: 7.4.1

Elasticsearch version: 7.4.1

APM Server version: 7.6.1

APM Agent language and version: java, apm-agent-attach-1.17.0.jar

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: docker

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup?
https://www.elastic.co/guide/en/apm/agent/java/current/setup-javaagent.html#setup-tomcat

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I have tomcat with serveral java web services (serveral war files) on it.
After setting up apm-agent in tomcat and apm-server, I can successfully send the data to kibana.
It all works well so far.

But when I need to get one of my war updated.
Here are my steps：

  • remove one war file from webapps, the other wars remains the same
  • send a new war (with the same name as the one I deleted) to webapps

With settings in server.xml

<Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">

I didn't restart my tomcat. The new war automatically get deployed and unpacked.
And the updated service works well.

But In kibana,
I can't receive data from the updated service anymore.
The other services still work fine.

2

Hi @bob96589, welcome to the forum.

Can you provide the DEBUG logs of the agent when doing the re-deploy operation ? (see documentation)

What versions of the JDK and Tomcat are you using ?

3

Hi @Sylvain_Juge. Thanks for your reply.

The following link is DEBUG logs of the agent, and it consists

  • apm-agent start
  • trigger service api (SpcAlphaParamController#getSpcAlpha)
  • apm-agent successfully send data to apm-server, and can see data in kibana
  • update war in tomcat
  • trigger service api again (SpcAlphaParamController#getSpcAlpha)
  • cannot see data in kibana

docker images: tomcat:9-jdk8
in the container java version is 1.8.0_252

APM Agent version: apm-agent-attach-1.16.0.jar (Sorry, in the previous post, I wrote the wrong version)

4

Hi @bob96589,

I was unable to reproduce the behavior that you observed locally with the following configurations:

  • agent 1.17.0 + tomcat 8.5.54 + OpenJDK 8u242
  • agent 1.16.0 + tomcat 9.0.37 + OpenJDK 8u242

In agent debug logs:

  • there is no error while sending data to apm-server
  • sample rate is at default value, thus all transactions should be captured
  • I can't really pinpoint the time where the re-deploy happens
  • the transaction seems to be still created after re-deploy (and we can see some SQL queries spans), but there is only a few of them.

Thus I have few other questions:

  • is this issue limited to transactions ? for example, are other metrics (like JVM metrics) still sent to Kibana after re-deploy ?
  • did you tried with multiple requests after the re-deploy ? (sounds really obvious but I'd better be safe). For example, if there was previously 100 transactions/s, do you see none of them after re-deploy even waiting for a few minutes ?
  • does restarting the whole JVM makes it work again ?
5

Hi @Sylvain_Juge,

Answer the question:

  • Is JVM metrics stored in apm-7.6.1-metric-* index? In apm-7.6.1-metric-* index, I can only find data with service.name equals to "tomcat-application". And even after re-deployed, the data is consistently being sent to the index.

  • Yes. I did try multiple requests. But the log in the previous post, I only did a successful and a fail request to avoid too much log.

  • Yes. Restarting the JVM (Restarting tomcat) fixs the problem.

Following link is a animated gif that demonstrates how I reproduce the problem.
The configuration is agent 1.16.0 + tomcat 9.0.37 + OpenJDK 8u242.

https://github.com/bob96589/temp/blob/master/error.gif

6

The end of the log you uploaded clearly shows the agent still traces SpcAlphaParamController#getSpcAlpha after redeployment and it even seems as it sends these data.
So, my suspicion is that there is a problem related to the service name of the redeployed app.

Please try the followings and provide feedback:

  1. Try setting agent's log_level to TRACE. This will print a lot of info, part of which is the full trace documents sent to the APM server. The communication with APM server relies on NDJSON, so please look for metadata JSONs sent before and after redeployment and upload them.
  2. It seems that the agent is using the auto-discovered service name. Please try setting the service_name config and see if the problem is resolved.
7

Hi @Eyal_Koren,

  1. Here is the log with log_level TRACE.

entire log: https://gist.github.com/bob96589/1b478f03785375fa04f1d9f4c4e7ebde
transaction log sent before deploy (extract from entire log): https://gist.github.com/bob96589/8165c00e70686e571378a2b06bdc8889
transaction log sent after deploy (extract from entire log):
https://gist.github.com/bob96589/82f9a5c40068b5302ee4c23d2cd5a6e2

I noticed a difference between the tranction logs being sent before and after redeploy.

Before redeploy, the json being sent to kibana consists service.name, "SPC-Service-1-0", which is an auto-dicovered service name. After data being sent to kibana, I can see the data in "SPC-Service-1-0" service.

After redeploy, service.name in the json was missing. (Probably it is a bug?!) After data being sent to kibana, I can see the data in "tomcat-application" service.

So No matter redeploy or not, the data is always being sent and can be received in kibana, but categorized in different service name.

  1. Setting the service_name config solves the problem and makes the data always stay in the right service name.
8

Extremely useful info, thanks!
I think I know what the problem is, will try a fix tomorrow

9

@bob96589 Please try this fix and let me know if it resolves the problem

10

Hi @Eyal_Koren,

Problem solved. Thanks.

11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.