Increase setCustomContext context character limit

[isurfbecause (Winston Yee)]

Kibana version:
6.5.4

Elasticsearch version:
6.5.4

APM Server version:
6.5.4

APM Agent language and version:
elastic-apm-node 2.5.1

Browser version:
Google Chrome
Version 73.0.3683.86 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Docker community images

Fresh install or upgraded from other version?
Fresh install

Is there anything special in your setup?
For testing purposes I am running my Node.js app locally that is sending metrics to the apm server behind an AWS ALB.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I am using express ^3.21.2 and trying to to send back the response body via the setCustomContext method. However the data seems to be truncated around character 578. I assume there is a max number of characters that can be sent. Is there a way to override the default limit. In the screenshot below take a look in responseData the characters are cut off at the key "fil (key should be filterField)

Here is the code I am using to send back the response.

app.use (function (req, res, next) {
  const oldWrite = res.write;
  const oldEnd = res.end;

  const chunks = [];

  res.write = (...restArgs) => {
    chunks.push(Buffer.from(restArgs[0]));
    oldWrite.apply(res, restArgs);
  };

  res.end = (...restArgs) => {
    if (restArgs[0]) {
      chunks.push(Buffer.from(restArgs[0]));
    }
    const body = Buffer.concat(chunks).toString('utf8');

    let customData = {
      time: new Date().toUTCString(),
      fromIP: req.headers['x-forwarded-for'] || 
      req.connection.remoteAddress,
      method: req.method,
      originalUri: req.originalUrl,
      uri: req.url,
      requestData: req.body,
      responseData: body,
      referer: req.headers.referer || '',
      ua: req.headers['user-agent']
    };

    apm.setCustomContext({
      responseBody: customData
    });
    
    console.eyes(customData);

    // console.log(body);
    oldEnd.apply(res, restArgs);
  };

  next();
});

Steps to reproduce:
Use the JS code above in your express app and send back large response (over 578 characters) to the setCustomContext method.

[wa7son (Thomas Watson)]

All strings under context.custom are truncated at 1024 bytes before being sent to the APM Server. (the only reason I can think of why it would truncate at a lower byte-size is if the string contains unicode characters).

For just capturing the HTTP body, you should instead use the captureBody config option. I think this is limited to a slightly higher 2048 bytes max length + you don't need to manually add the body as custom data.

There's currently no way to override the max limit of 1024 bytes for context.custom strings. To do that you'd have to add the following line to the sanitize config:

        truncateStringsAt: <new-limit>,

This however would influence many of the other strings that are sent to the APM Server and not only under custom.context.

The reason why we limit these is that it would otherwise fill up the index in Elasticsearch and would require more bandwidth between the agent and APM Server. Is this is a one-off need, or is this a use-case that you need normally?

[isurfbecause (Winston Yee)]

Thank you for the reply.

Yes, it is strange to truncate on the key name filter that has no unicode characters.

I was looking at the docs for captureBody but that targets the request body instead of the response body. We are interested in the response body but I am sure as a last option we could attach the response body to the request body.

This is not a one-off need, we think the response body would be great for debugging. For example, a specific user was receiving an error. We searched the transactions by user's email (awesome!) but we needed a bit more context in the response such as the response body. If there is a different approach on gaining debugging incites please do share. Thank you very much

[wa7son (Thomas Watson)]

I'll discuss this with the rest of the APM team to see if this is something we can add support for. I'll keep you posted

[isurfbecause (Winston Yee)]

Thanks Thomas.