Elastic APM Java - Transactions & Spans are recorded but not reported to Elastic APM Server or Kibana

[Debashish_BHARALI]

I have a standalone JAVA application.

And have integrated it successfully with Elastic APM (+ElasticSearch +Kibana) for capturing telemetries.

Java Version: 8 - OpenJDK

Elastic Agent & Library Version: 1.16

Elastic Search, APM and Kibana Version: 7.7.1

Below are the relevant JVM Options being used:

JAVA_OPTS="$JAVA_OPTS -javaagent:$BASE_HOME/agent-lib/elastic-apm-agent-1.16.0.jar -Delastic.apm.service_name=my-app -Delastic.apm.server_urls=http://elastic-apm-server:8200"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.application_packages=com,org -Delastic.apm.span_frames_min_duration=-1ms"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.log_file=$BASE_HOME/logs/apm.log -Delastic.apm.log_level=DEBUG"

• I am generating custom transactions and spans using the Tracer/Transaction/Span APIs as suggested in the official documentation.
• And as per the generated debug logs. These spans and transactions are getting captured as expected.
• I have validated the same by DEBUGGING it over the IDE, that transactions are being captured as expected.

Problem: The custom transactions are not shown on the Kibana APM Dashboard

However some out of the box transactions from Quartz(which is being used in the application) are shown as expected. Which should mean the integration with the Elastic APM Server is fine.

It appears to me, even though the transactions are being captured successfully, those are not reported(sent) to the APM Server

Refer some relevant apm logs:

2020-07-01 12:33:09.569 [pool-1-thread-1] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction '' 00-d0025079170e4f03698702f4e68be4ac-cf792454fbef1c77-01 (16970dc3) {
2020-07-01 12:33:09.569 [pool-1-thread-1] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Activating 'ExtractionRequestHandler#invokeExtraction' 00-d0025079170e4f03698702f4e68be4ac-cf792454fbef1c77-01 (16970dc3) on thread 26
2020-07-01 12:33:09.569 [pool-1-thread-1] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to 'ExtractionRequestHandler#invokeExtraction' 00-d0025079170e4f03698702f4e68be4ac-cf792454fbef1c77-01 (16970dc3) (2)
2020-07-01 12:33:09.569 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 86)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'ExtractionRequestHandler#invokeExtraction' 00-98a1d8f4970d585915eb03a414b7b14c-994dd2823198f1ef-01 (33d448b5) (4)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'BOpFileUtils#authorizeFilePath' 00-98a1d8f4970d585915eb03a414b7b14c-133200d1793fbaab-01 (67fba8aa) (0)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 87)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'ExtractionRequestHandler#invokeExtraction' 00-98a1d8f4970d585915eb03a414b7b14c-994dd2823198f1ef-01 (33d448b5) (3)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'SCR#init' 00-98a1d8f4970d585915eb03a414b7b14c-77cf207c33eb24ab-01 (2f1f25c3) (0)

Need help in finding what I am doing wrong? And how to fix it?

[felixbarny (Felix Barnsteiner)]

Hi and welcome to the forum

You might have to select a different transaction type here:

[Debashish_BHARALI]

Thanks a ton @felixbarny.

That was the problem.
I wasted many hours on this.

In my case, the TYPE in which all my custom transactions are coming is with name 'Custom'.
Can we somehow change the type somewhere?

[felixbarny (Felix Barnsteiner)]

Sorry for the inconvenience. You can use the API to override the type.
Or are you suggesting the default should be different?
I’m also wondering whether the UI should just show all types by default with the option to select only specific ones. What do you think?

[Debashish_BHARALI]

@felixbarny

What

Default value of Custom is perfect.
It was my fault that I did't go through the drop down.

API to override? --> are you pointing to span.setType()? If yes, I observed that is deprecated.

Yes I agree. Personally, as a user, I think following should be there:

1. User should be able to select a default filter. I.e. Any of the available filter can be marked as default for the user (user account for which it is logged in)
2. For the selected default filter (currently Transaction Type), X numbers of available options should be visible (may be 5 or 10) [may be with pagination]
3. Even the default filter (currently Transaction Type) should be a multi-select like Host, Container ID etc

[felixbarny (Felix Barnsteiner)]

It would be Transaction.setType

[Debashish_BHARALI]

@felixbarny Got it.

[felixbarny (Felix Barnsteiner)]

@Debashish_BHARALI

Yes I agree. Personally, as a user, I think following should be there:

1. User should be able to select a default filter. I.e. Any of the available filter can be marked as default for the user (user account for which it is logged in)
2. For the selected default filter (currently Transaction Type), X numbers of available options should be visible (may be 5 or 10) [may be with pagination]
3. Even the default filter (currently Transaction Type) should be a multi-select like Host, Container ID etc

That’s great feedback for the UI team!
cc @sqren