Transaction for application startup is added to default service name

[Wolfram_Haussig]

Hello all,

I have an application which takes a very long time to startup(depending on the circumstances between 50 seconds and 5 minutes). Therefore, I have added code to instrument the startup(I have subclassed the Spring XmlWebApplicationContext and added @CaptureTransaction to the relevant methods like refresh). This worked - and is still working.

Now, I have upgraded the apm agent to the latest version and wanted to try the service name auto discovery to separate the applications within our Tomcat installation. I am aware that the Metrics are sent to the default service name(in my case tomcat-application) but I found that not only the metrics are being sent there - my startup transactions are stored there too. This is unfortunate for me because naturally, we would be looking in the application for the startup transaction.

My guess is that the apm agent does not know about the application as it is still starting - does anyone have an idea how to work around that?

Elasticsearch version: 7.10

APM Server version: 7.10

APM Agent language and version: java, 1.20.0

Best regards
Wolfram

[felixbarny (Felix Barnsteiner)]

Hello again

Hmm, interesting issue... Initially, I was thinking that we could move the logic to determine the service name to the end of the transaction as opposed to the start. But that doesn't work as the service name is copied to each span of the transaction. As spans are sent right after the corresponding event occurs, we can't can't change the transaction's service name anymore.

As a workaround, could you add a custom label to the transaction?

[Wolfram_Haussig]

Hello to you too Barny! You know, I always have intersting issues

@felixbarny

As a workaround, could you add a custom label to the transaction?

I could do that for sure. I am just unsure if I should do that because I would have to switch our infrastructure:

• send the APM data to Logstash instead of Elasticsearch
• change the service name to the label value if the label exists
• do that for the spans too
• ...

Or did I missunderstand you?

Best regards
Wolfram

[felixbarny (Felix Barnsteiner)]

The workaround with the labels would not a proper workaround. The startup transactions would still be in the tomcat-application service name, unfortunately. I currently can't think of any proper way. Even when using logstash or ingest pipelines, it probably wouldn't be doable as the spans are sent separately from their transactions.

I think what I was trying to ask is how severe that limitation is. It's definitely annoying and counter-intuitive that it's not listed in the same service but once you know where to look for these you get what you need, right?

Which service name auto-detection are you relying on? The <display-name> from web.xml or the spring.application.name from Spring's application.properties?

[Wolfram_Haussig]

@felixbarny

once you know where to look for these you get what you need, right?

Yes, that is right and I will take this route if no other way can be found but I would still be glad to have another way.
Is there a possibility to set a label on a transaction which will be inherited by all spans so I could transfer them by logstash or ingest pipeline?

@felixbarny

Which service name auto-detection are you relying on?

I am relying on the web.xml detection.

[felixbarny (Felix Barnsteiner)]

@Wolfram_Haussig

Is there a possibility to set a label on a transaction which will be inherited by all spans so I could transfer them by logstash or ingest pipeline?

No there is not. That would also have the issue that spans that have already started would not be aware of that label.

@Wolfram_Haussig

I am relying on the web.xml detection.

Is using spring.applicaiton.name an option? It might be that auto-detecting that happens early enough the initialization process.
For Spring, we're instrumenting WebApplicationContext#initPropertySources (see also https://github.com/elastic/apm-agent-java/blob/master/apm-agent-plugins/apm-spring-webmvc-plugin/src/main/java/co/elastic/apm/agent/springwebmvc/SpringServiceNameInstrumentation.java). I'm not sure if that's called before refresh but it might be.

For Servlet, we only detect the display-name when the first request gets executed, which is obviously after the application startup.

[Wolfram_Haussig]

@felixbarny

Is using spring.applicaiton.name an option?

Unfortunately, we are not using Spring Boot so this file is not used in our project. I tried setting this in our property file but this was not used.

Anyway - thanks a lot for your help!