CaptureTransaction not showing all spans in webflux project

[gfan8w]

I have a sample application, it is a WebFlux SpringBoot application (SpringBoot version=2.5.3, WebFlux version=5.3.9) embedded an Elastic APM agent (version=1.28.0). Start the Spring WebFlux app:
mvn spring-boot:run -Dspring-boot.run.arguments=--server.port=8085

go to the docker folder: cd ../elastic-apm-java/docker , run docker-compose up -d to set up a dev APM server.

Do some controller accessing. then go the Kibana/APM portal.
I found that the Transaction or Span are not recored.

But, I tried another Spring MVC app: elastic-apm-java. After running the app, the go to the Kibana/APM portal, I can see the Transaction or Span are recored.

I am wondering where is the WebFlux APM wrong? Is the version of ES & Kibana not correct? Mismatched with APM agent?
My sample code is here.

Update-1： I have enable the experimental feature in my code.
file: elasticapm.properties under the resources folder, value is: enable_experimental_instrumentations=true.
When the app start up, it prints log:

[Attach Listener] INFO co.elastic.apm.agent.configuration.StartupInfo - enable_experimental_instrumentations: 'true' (source: Attachment configuration)

So, I think the enable_experimental_instrumentations is in effect.

Update 2:
I modify the docker-compose.yaml, now the user-microservice and user-webfluxservice can run in docker, just start the docker-compose file, then observe the APM dashboard.

It is slightly different, it can display the random CaptureSpan, but the offset position is the same as the UserController#all, it's not correct.

the corresponding code is:

@CaptureSpan("random")
    public int random(int min, int max) {

        return new Random().nextInt(max - min) + min;
    }

[Sylvain_Juge]

Hi @gfan8w , the Webflux instrumentation is currently in experimental status, so you have to set enable_experimental_instrumentations=true otherwise it gets disabled by default (see documentation: Core configuration options | APM Java Agent Reference [1.x] | Elastic).

We are planning to enable it by default for the next release, so it won't be necessary in the future.

[gfan8w]

I have updated my question. I have already enabled the enable_experimental_instrumentations=true, I forgot to mention it.

[Eyal_Koren]

1.28.0 is quite old and this particular version suffered from some issues.
So first thing to do is try the latest version (still with enable_experimental_instrumentations=true).

If that still doesn't work, please set log_level=debug in your WebFlux app, restart and invoke a couple of requests and see if you can get a hint from the log. If possible, please share the entire log from startup and until after a few requests were handled.

[gfan8w]

I upgrade the APM agent client to 1.33. and the dashboard on the server side is still not show information correctly. I append full debug log here

The log:

log by user code: saved the user

is the info hit by user operation.

[Eyal_Koren]

The shared log is info level, thus doesn't contain a lot of info.
You'd need to modify the log_level config option for that.
It would be quite verbose, try to dig in it for possible clues.

[gfan8w]

Hi,
I update the debug log, the print an error:

Cannot resolve type description for javax.servlet.ServletContext

I searched for a while, it seems that this kind of error should not appear again, because now the the APM agent have support the WebFlux .

I also trigger the controller to make some data record by APM agent.
this log is printed by user accessing:

log by user code: saved the user:onComplete

I am new, I investigate the debug log, I could not find any reason why the data displayed on the Kibana portal is not correct.

[gfan8w]

I would like to post another question, I mentioned in the Update2:

I modify the docker-compose.yaml, now the user-microservice and user-webfluxservice can run in docker, just start the docker-compose file, then observe the APM dashboard.

the user-microservice(mvc) and user-webfluxservice(webflux) can run in docker or just the dev host machine.

This is the event of user-microservice(mvc) running on the dev host machine (JDK11)

This is the event captured of user-microservice(mvc) running in the docker(JDK11):

The code are the same.

public User save(User user) {
        sleep();    // sleep has @CaptureSpan("otherOperations"), so it should display a span on the Kibana portal
        return userRepository.save(user);
    }

[Sylvain_Juge]

Hi,

We just released version 1.34.0 yesterday (release notes), the Webflux instrumentation is enabled by default.

Thus I suggest doing the following to debug your issue:

• update to agent version 1.34.0
• start the agent with log_level=info (or an higher log-level) and compare the first few lines of the agent logs when the agent starts for any difference in agent configuration.

If there is any difference in configuration with a previous agent version it could have been the likely issue, but with 1.34.0 Webflux should be enabled and not depend anymore on configuration.

[gfan8w]

Does elastic APM support r2dbc in WebFlux? From here, it seems still not support r2dbc?

[Sylvain_Juge]

Hi @gfan8w ,

As you have found, there is currently no support for r2dbc in Elastic APM agent, there hasn't had many requests for it thus it is not currently planned. However it is still possible to create an internal/external plugin for it, or try using manual instrumentation through a Tracing API.

[gfan8w]

Thanks for your reply. now, I think I know why the span data is not showing on the webpage. I should write tracing code to record more information.

Thank you!