Elastic APM Java Agent configuration for logs in docker container

[_Arnaud]

Hello,

I am testing the elastic stack (8.6.0) with kibana (8.6.0) and elastic agent

I have a question regarding the elastic APM java agent 1.35.0.

I have docker container running a java app and I have started the java app with the following JAVA option in order to send my application performance to elastic:

-javaagent:/tmp/elastic-apm-agent-1.35.0.jar -Delastic.apm.service_name=myapp -Delastic.apm.application_packages=com.company.myapp -Delastic.apm.server_url=http://xxx.xxx.xxx.xxx:8200 -Delastic.apm.secret_token=

This is working almost fine (In kibana observability, I can see my apm service overview, transactions, dependencies,errors...) which is really good !!
But I cannot see any log. I guess it comes from the fact that I am following the 12 factors and I send my java logs to STDOUT.
Is there a way to retrieve those logs using the apm java agent which runs inside my container (without modifying my source code) ?

Thanks and regards,

Arnaud

[Kevin_Delemme]

Hi @_Arnaud,

Are you using one of the supported logging framework or just System.out ?
If you follow the setup guide for log correlation, and make sure you are using one of the supported logging framework, the logs should be made available.

You might need to change your source code if you're not using a logging framework supported. Or decide to ship the logs to file, and then use filebeat to ship the log.

Thanks,
Kevin

[Sylvain_Juge]

For completeness, we just added a new section to explain the possible log ingestion strategies in the documentation:

Application logs | Observability Guide [master] | Elastic

I'ts in main branch, but it also applies to already released versions.

In you case, using the APM agent ECS reformatting option could be relevant, it allows to reformat your application logs directly to ECS format without modifying the application code.
Once this is done, all you have to do is to add the filebeat to ingest the ECS reformatted logs.

The Java agent will soon be able to send the logs directly to apm-server, thus ingestion could be done without filebeat.

[_Arnaud]

Hi Sylvain, Kevin,

Thank you for you answers, you are right, I have to install filebeat to retrieve logs.
The documentation mislead me.
I'm waiting for the release of version 1.36 of apm server in order not to have to install multiple client for sending logs/apm/metrics.

This topic can be closed.

[Sylvain_Juge]

Hi @_Arnaud , it is the Java agent version 1.36.0 (which will likely be available soon) that will include the ability to send logs. It also requires 8.6.0 or later for apm-server and other parts of the stack.

Please note that there are a few limitations with log sending through the Java agent, as it is an experimental feature for now (which just means it might evolve a bit before being GA). For example, the log attributes are not all processed as they would through the usual filebeat ingestion.

On a related note, I have started creating an end-to-end example of log ingestion when combined with the Java agent, so you'll soon have a few examples to help you setup your app: ingest logs with java app by SylvainJuge · Pull Request #65 · elastic/apm-contrib · GitHub, as I am writing this, it currently contains plaintext log file ingestion with filebeat (which I don't recommend for simplicity), but will also contain ECS logging (with an extra logger dependency in the app), ECS reformatting (with the agent) and log sending.

[Lukefilewalker (Lucas Senn)]

Thank you very much for the explanation @Sylvain_Juge. The current documentation confused me also as I thought it should be possible to include all logs via the java apm agent.

Do you know when 1.36.0 will be released and/or is there a possibility to access SNAPSHOT Releases for testing purposes?

I love the linked Pull Request #65, as it was the missing piece, which helped me to understand how to integrate logs also via filebeat from Spring. Thank you very much for sharing. As I am using the filebeat.autodiscover with type:kubernetes, and hints.enabled: true I had to annotate my container with the following hints to get filebeat correctly translate the ECS Formatted Logs:

co.elastic.logs/json.keys_under_root: "true"
co.elastic.logs/json.message_key: "message"
co.elastic.logs/json.overwrite_keys: "true"

Placing it here if someone also stumbles upon this question and needs a hint.

[Sylvain_Juge]

@Lukefilewalker there is a link in the readme but it's broken, you can use the URL used in the Dockerfile in the mean time to get it from the CI: apm-contrib/Dockerfile at a88ad385e1174d12b5828e7bb5a2fc1678766ec6 · elastic/apm-contrib · GitHub.