# APM capture body is broken after upgrading from 7.14.0 to 7.15.0

**URL:** <https://discuss.elastic.co/t/apm-capture-body-is-broken-after-upgrading-from-7-14-0-to-7-15-0/285300>\
**Category:** APM\
**Tags:** java, server\
**Created:** [September 28, 2021, 4:49am UTC](https://discuss.elastic.co/t/apm-capture-body-is-broken-after-upgrading-from-7-14-0-to-7-15-0/285300 "2021-09-28T04:49:44Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ali\_Nazemian](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@Ali\_Nazemian](https://discuss.elastic.co/u/Ali_Nazemian)\
**Post date:** [September 28, 2021, 4:49am UTC](https://discuss.elastic.co/t/apm-capture-body-is-broken-after-upgrading-from-7-14-0-to-7-15-0/285300/1 "2021-09-28T04:49:44Z")

</div>

Since I have upgraded to APM server 7.15.0 (on Elastic cloud) when capture\_body is enabled JSON body is stored in a single field. This is not different from the older versions. However, the value of JSON message stored is not normalized which means if there are any characters such as quotations in that field, it must be escaped. This is used to work in 7.14.x but seems to be broken after it's upgraded.

Example of an APM message in 7.14.0

```auto
{
  "http": {
    "request": {
      "method": "PUT",
      "body.original": "[{\"channel_id\":\"123456\",\"channel_name\":\"Test\",\"source_username\":null,\"url\":\"xyz.com\"}]"
    }
  }
}

```

Same message in 7.15.0:

```auto
{
  "http": {
    "request": {
      "method": "PUT",
      "body.original": """[{"channel_id":"123456","channel_name":"Test","source_username":null,"url":"xyz.com"}]"""
    }
  }
}

```

A workaround could be setting an ingest pipeline to process documents and escape all required characters but I am not sure which processor would give me that option.

P.S: I am using java-agent (1.26.0) with Spring boot 2.5.4

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 28, 2021, 6:13am UTC](https://discuss.elastic.co/t/apm-capture-body-is-broken-after-upgrading-from-7-14-0-to-7-15-0/285300/2 "2021-09-28T06:13:51Z")

</div>

Where do you get that JSON from? Did you copy it from a query in Kibana?  
I thing that's just a different way the JSON is quoted in the presentation layer to make it easier to ready or copy/paste. Note that in 7.15, the string starts with three quotes (`"""`) this makes escaping the quotes in the JSON (`\"`) unnecessary.  
If you directly query Elasticsearch, I'd expect the same output as in 7.14.

---

<div class="post-metadata">

**Author:** ![Ali\_Nazemian](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@Ali\_Nazemian](https://discuss.elastic.co/u/Ali_Nazemian)\
**Post date:** [September 28, 2021, 7:14am UTC](https://discuss.elastic.co/t/apm-capture-body-is-broken-after-upgrading-from-7-14-0-to-7-15-0/285300/3 "2021-09-28T07:14:44Z")

</div>

I got JSON from query Elastic APM index directly (not using Kibana). The reason I am suspicious about this is that there is a separate ingest pipeline processor to parse JSON for `body.original`  
, but that is not working so I thought this is because the String representation of that JSON field is not valid anymore.

```auto
PUT _ingest/pipeline/apm_parse_body
{
  "description": "Parse APM message body content",
  "processors": [
    {
      "json": {
        "if": "boolean originalIsMap = ctx.http?.request?.body?.original instanceof Map; ctx.http?.request?.body?.original != null && ctx.http?.request?.body?.original != '[REDACTED]' && !originalIsMap",
        "field": "http.request.body.original"
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 28, 2021, 9:46am UTC](https://discuss.elastic.co/t/apm-capture-body-is-broken-after-upgrading-from-7-14-0-to-7-15-0/285300/4 "2021-09-28T09:46:11Z")

</div>

I found out what the issue is.

In 7.15, there was a change ([Consolidate model.HTTP and model.Http by axw · Pull Request #5764 · elastic/apm-server · GitHub](https://github.com/elastic/apm-server/pull/5764)) where the `body.original` field turned from a nested field to a dotted field.  
This breaks ingest node pipelines where every field is expected to be nested. I'd consider this a bug in APM Server.

I have a workaround for you, though. You can use the `dot_expander` processor to convert the dotted field name into a nested one:

```auto
PUT _ingest/pipeline/apm_parse_body
{
  "description": "Parse APM message body content",
  "processors": [
    {
      "dot_expander": {
        "path": "http.request",
        "field": "body.original"
      }
    },
    {
      "json": {
        "if": "ctx.http?.request?.body?.original instanceof String && ctx.http?.request?.body?.original != '[REDACTED]'",
        "field": "http.request.body.original"
      }
    }
  ]
}
DELETE /apm-test?ignore_unavailable=true
POST /apm-test/_doc?pipeline=apm_parse_body&refresh=true
{
  "http": {
    "request": {
      "method": "PUT",
      "body": {
        "original": """[{"channel_id":"123456","channel_name":"Test","source_username":null,"url":"xyz.com"}]"""
      }
    }
  }
}
POST /apm-test/_doc?pipeline=apm_parse_body&refresh=true
{
  "http": {
    "request": {
      "method": "PUT",
      "body.original": """[{"channel_id":"123456","channel_name":"Test","source_username":null,"url":"xyz.com"}]"""
    }
  }
}
POST /apm-test/_search

```

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 28, 2021, 9:50am UTC](https://discuss.elastic.co/t/apm-capture-body-is-broken-after-upgrading-from-7-14-0-to-7-15-0/285300/5 "2021-09-28T09:50:10Z")

</div>

I have created an APM Server issue for that: [Dotted field `http.request.body.original` breaks ingest processors · Issue #6255 · elastic/apm-server · GitHub](https://github.com/elastic/apm-server/issues/6255)

---

<div class="post-metadata">

**Author:** ![Ali\_Nazemian](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@Ali\_Nazemian](https://discuss.elastic.co/u/Ali_Nazemian)\
**Post date:** [September 28, 2021, 11:36am UTC](https://discuss.elastic.co/t/apm-capture-body-is-broken-after-upgrading-from-7-14-0-to-7-15-0/285300/6 "2021-09-28T11:36:05Z")

</div>

Great. Thank you. I have checked the workaround and it's fine. I hope the main issue will be resolved soon so I can remove the dot\_expander processor. Thanks for your help.

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [September 28, 2021, 2:45pm UTC](https://discuss.elastic.co/t/apm-capture-body-is-broken-after-upgrading-from-7-14-0-to-7-15-0/285300/7 "2021-09-28T14:45:56Z")

</div>

Thanks for the investigation @felixbarny. I've put up a PR [https://github.com/elastic/apm-server/pull/6256](https://github.com/elastic/apm-server/pull/6256).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2021, 10:46am UTC](https://discuss.elastic.co/t/apm-capture-body-is-broken-after-upgrading-from-7-14-0-to-7-15-0/285300/8 "2021-10-19T10:46:32Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.