Displaying custom errors in Kibana APM using java agent

[Sefra]

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version: 6.8.5

Elasticsearch version: 6.8.5

APM Server version: 6.8.11

APM Agent language and version: 1.19.0

Browser version: Vivaldi 3.5.2115.81

Original install method (e.g. download page, yum, deb, from source, etc.) and version: install from docker hub, versions are above

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

I am using nomad for orchestration (it's basically like kubernetes) of my services written in java and using spring boot. It's all running in docker containers, the apm server is running there, too. The logs of the containers are automatically shipped to ES using filebeat.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I have custom application errors. These include technical errors like no database connection or other services not available. There are also functional errors, like data problems in the database, false formatting and other things.
I installed elastic apm and really like the graphical representation in Kibana. It works fine, I can see detailed information e.g. for REST endpoints that are called in my services, how long things take etc.
What I would also like to see is the errors of my services in the errors tab. Currently I only have one error in the errors tab which is an internal exception that my service didn't catch by itself. However all the errors that were generated by the application are not visible.
But I would really like to see them there, since the UI provides some really nice features that I want to use in the future to get a better overview of the errors.

The errors are created in the line
log.error(exception.getMessage(),exception)

and I can see them in the application log index in ES, so they get transferred without a problem. But the apm agent doesn't seem to see them as an error, so I can't find them in Kibana.

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

[Eyal_Koren]

Hi @Sefra, welcome to our forum

If you use slf4j version 1.4.1 or higher or if you are using log4j2, than this will be automatically captured. If you are using something else, then these are the two immediate options I can think of:

1. Add slf4j as your logging abstraction layer, while still using your current logging framework
2. Use our public API, which allows to capture any Exception as an error in the right context of the span/transaction active at the time of error occurrence.

I hope this helps!

[Sefra]

@Eyal_Koren I have the log correlation activated. When a new log entry is created (be it an info or error message) I get the

trace.id

and transaction.id in the mdc block and can see the messages in Kibana APM just fine, that works. The messages are also visible in Kibana logs. However, when I have an error message, there is no error.id in the mdc block. But according to your page, this should be there.
logback-classic 1.2.3 and slf4j 1.7.30 are used for logging.
So the versions shouldn't be a problem. Is there something else I could check on why the call is not recognized as an error message?!

[Eyal_Koren]

OK, then everything you describe here should be supported.
The problem seems to be that our instrumentation is not applied to the slf4j loggers for some reason. Because of that, the agent does not create APM Error events, which also explains why there is no error.id in the MDC data.

Please set log_level=DEBUG and upload the resulted log. Especially, we are looking for log entries saying Type match for instrumentation Slf4jLoggerErrorCapturingInstrumentation and Method match for instrumentation Slf4jLoggerErrorCapturingInstrumentation.

Also, it could be useful if you share the entire agent configuration you are using, as well as the code that logs the Exception that you want to see captured.

Thanks.

[Sefra]

Thank you for your help! I was able to fix the problem. After talking to the dev, it turned out that the exception class didn't inherit from any of the Java generic exception classes like Throwable or Exception. After changing this, it now works and I can see the errors in Kibana. Problem solved

[Eyal_Koren]

These are great news! Thanks for updating.