APM capture body is broken after upgrading from 7.14.0 to 7.15.0

[Ali_Nazemian (Ali)]

Since I have upgraded to APM server 7.15.0 (on Elastic cloud) when capture_body is enabled JSON body is stored in a single field. This is not different from the older versions. However, the value of JSON message stored is not normalized which means if there are any characters such as quotations in that field, it must be escaped. This is used to work in 7.14.x but seems to be broken after it's upgraded.

Example of an APM message in 7.14.0

{
  "http": {
    "request": {
      "method": "PUT",
      "body.original": "[{\"channel_id\":\"123456\",\"channel_name\":\"Test\",\"source_username\":null,\"url\":\"xyz.com\"}]"
    }
  }
}

Same message in 7.15.0:

{
  "http": {
    "request": {
      "method": "PUT",
      "body.original": """[{"channel_id":"123456","channel_name":"Test","source_username":null,"url":"xyz.com"}]"""
    }
  }
}

A workaround could be setting an ingest pipeline to process documents and escape all required characters but I am not sure which processor would give me that option.

P.S: I am using java-agent (1.26.0) with Spring boot 2.5.4

[felixbarny (Felix Barnsteiner)]

Where do you get that JSON from? Did you copy it from a query in Kibana?
I thing that's just a different way the JSON is quoted in the presentation layer to make it easier to ready or copy/paste. Note that in 7.15, the string starts with three quotes (""") this makes escaping the quotes in the JSON (\") unnecessary.
If you directly query Elasticsearch, I'd expect the same output as in 7.14.

[Ali_Nazemian (Ali)]

I got JSON from query Elastic APM index directly (not using Kibana). The reason I am suspicious about this is that there is a separate ingest pipeline processor to parse JSON for body.original
, but that is not working so I thought this is because the String representation of that JSON field is not valid anymore.

PUT _ingest/pipeline/apm_parse_body
{
  "description": "Parse APM message body content",
  "processors": [
    {
      "json": {
        "if": "boolean originalIsMap = ctx.http?.request?.body?.original instanceof Map; ctx.http?.request?.body?.original != null  && ctx.http?.request?.body?.original != '[REDACTED]' && !originalIsMap",
        "field": "http.request.body.original"
      }
    }
  ]
}

[felixbarny (Felix Barnsteiner)]

I found out what the issue is.

In 7.15, there was a change (Consolidate model.HTTP and model.Http by axw · Pull Request #5764 · elastic/apm-server · GitHub) where the body.original field turned from a nested field to a dotted field.
This breaks ingest node pipelines where every field is expected to be nested. I'd consider this a bug in APM Server.

I have a workaround for you, though. You can use the dot_expander processor to convert the dotted field name into a nested one:

PUT _ingest/pipeline/apm_parse_body
{
  "description": "Parse APM message body content",
  "processors": [
    {
      "dot_expander": {
        "path": "http.request",
        "field": "body.original"
      }
    },
    {
      "json": {
        "if": "ctx.http?.request?.body?.original instanceof String && ctx.http?.request?.body?.original != '[REDACTED]'",
        "field": "http.request.body.original"
      }
    }
  ]
}
DELETE /apm-test?ignore_unavailable=true
POST /apm-test/_doc?pipeline=apm_parse_body&refresh=true
{
  "http": {
    "request": {
      "method": "PUT",
      "body": {
        "original": """[{"channel_id":"123456","channel_name":"Test","source_username":null,"url":"xyz.com"}]"""
      }
    }
  }
}
POST /apm-test/_doc?pipeline=apm_parse_body&refresh=true
{
  "http": {
    "request": {
      "method": "PUT",
      "body.original": """[{"channel_id":"123456","channel_name":"Test","source_username":null,"url":"xyz.com"}]"""
    }
  }
}
POST /apm-test/_search

[felixbarny (Felix Barnsteiner)]

I have created an APM Server issue for that: Dotted field `http.request.body.original` breaks ingest processors · Issue #6255 · elastic/apm-server · GitHub

[Ali_Nazemian (Ali)]

Great. Thank you. I have checked the workaround and it's fine. I hope the main issue will be resolved soon so I can remove the dot_expander processor. Thanks for your help.

[simitt (Silvia Mitter)]

Thanks for the investigation @felixbarny. I've put up a PR https://github.com/elastic/apm-server/pull/6256.