APM Agent language and version: Java 1.24.0
Browser version: Chrome 91
Hello all,
I am still testing the Java APM agent with log_ecs_reformatting
enabled and found the following problems:
- APM agent generates (for ElasticSearch) invalid JSON objects when adding an ECS field to the MDC which is also generated by the Agent
- use the following code to generate a log
MDC.put("event.dataset", "security");
log.info("test");
- this is the JSON message
{
"@timestamp": "2021-06-23T08:53:45.345Z",
"log.level": "INFO",
"message": "test",
"ecs.version": "1.2.0",
"service.name": "tomcat-application",
"event.dataset": "tomcat-application.[redacted]",
"process.thread.name": "http-nio-2000-exec-8",
"log.logger": "[redacted]",
"transaction.id": "d05993a768800728",
"trace.id": "4b50419e4c42971617afc8cdfbe179e2",
"event.dataset": "security"
}
- this is the ingestion response
{
"error" : {
"root_cause" : [
{
"type" : "mapper_parsing_exception",
"reason" : "failed to parse"
}
],
"type" : "mapper_parsing_exception",
"reason" : "failed to parse",
"caused_by" : {
"type" : "json_parse_exception",
"reason" : "Duplicate field 'event.dataset'\n at [Source: (byte[])...[truncated 163 bytes]; line: 11, column: 20]"
}
},
"status" : 400
}
- proposal: do not store the MDC fields in the root and create an MDC subelement like this
{
"@timestamp": "2021-06-23T08:53:45.345Z",
"log.level": "INFO",
"message": "test",
"ecs.version": "1.2.0",
"service.name": "tomcat-application",
"event.dataset": "tomcat-application.[redacted]",
"process.thread.name": "http-nio-2000-exec-8",
"log.logger": "[redacted]",
"transaction.id": "d05993a768800728",
"trace.id": "4b50419e4c42971617afc8cdfbe179e2",
"mdc": {
"event.dataset": "security"
}
}
- Although the
co.elastic.logging.logback.EcsEncoder
supports storing the markers the APM agent does not store them asco.elastic.apm.agent.logback.LogbackEcsReformattingHelper
always setsecsEncoder.setIncludeMarkers(false);
increateEcsFormatter
- use the following code to create a log with a marker
log.info(MarkerFactory.getMarker("CONFIDENTIAL"), "test");
- use the following code to create a log with multiple markers
Marker dupMarker = MarkerFactory.getDetachedMarker("COMPOSITE_MARKER");
dupMarker.add(MarkerFactory.getMarker("SECURITY"));
dupMarker.add(MarkerFactory.getMarker("CONFIDENTIAL"));
log.info(dupMarker, "test");
- the markers should be shown in the response
- proposal: Either store the markers always or add a setting with which this feature can be disabled (I would expect the default to be enabled).
Best regards
Wolfram