Source IP and other metadata not present in logs when log_sending=true

Elastic Observability APM
1

Elastic Cloud version: 8.7.1

APM Agent language and version: Java, 1.52.0

Spring Boot version 2.7

Java version: 17

Logging Framework: SLF4J, both Logback and Log4j2 tested with same issue

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
We are using the Java APM agent with log_sending=true. This largely works fine, but some metadata is not added to the the majority of log messages. Examples:

  1. source.ip
  2. client.geo.* - none of these fields are included
  3. url.* - none of these fields are included
  4. user.* - none of these fields are included

The odd thing is that these attributes are populated for a few log messages. These fully enriched log messages seem to be triggered by Spring/Tomcat rather than our own code and have the pattern:

[][access] 1.1.1.1 someuser@example.com "GET /some_url? HTTP/1.1" 401

The noteworthy differences to normal log messages being the first empty square brackets and the second ones with [access]. It is also always for an error.

The question is: Is it possible to configure our "normal" log messages to contain all the missing attributes? They would be extremely useful for troubleshooting and the creation of alerts.

Thanks!

2

Log correlation only adds APM-related fields, (e.g. trace-id, span-id, service name).
The other fields you mentioned don't come from the agent, but are populated by your application itself (e.g. tomcat) and placed in the MDC.

Unfortunately, the log sending feature does currently not support sending additional values from the MDC, see the docs:

Log sending does not currently support custom MDC fields, log_ecs_reformatting and shipping the logs with Filebeat must be used if custom MDC fields are required.

The message where you see those fields are likely cases where tomcat/spring explicitly add those values to the message from the MDC.