Not able to see request body for errors in NodeJS elastic-apm agent for express framework

Kibana version: 8.8.1

Elasticsearch version: 8.8.1

APM Server version: 8.8.1

APM Agent language and version: NodeJs Agent elastic-apm-node": "^3.47.0

Fresh install or upgraded from other version? Upgraded from 7.17 recently

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I have added captureBody: 'errors' configuration in my express app, but on APM UI with 500 error it shows **http.request.body.original** [REDACTED]
Not sure where and how to see request body? as per documentation request body is there in req.body() as well still its not showing. I am not filtering out any sensitive information as well.

express index.js

const apm = require('elastic-apm-node').start({
  // Override the service name from package.json
  // Allowed characters: a-z, A-Z, 0-9, -, _, and space
  serviceName: 'example-service',

  // Use if APM Server requires a secret token
  secretToken: '<token>',

  // Set the custom APM Server URL (default: http://localhost:8200)
  serverUrl: '<url>',
  ignoreUrls: ['/', '/metrics'],
  captureBody: 'errors'
});

I have also checked Error tab but there are no traces there as well.

APM UI Screenshot for metadata

@Kesha_Shah Thanks for the question.

I think the confusion is whether the captured body is shown on the "transaction" object/document or the "error" object/document. When an Express handler errors, e.g.:

app.post('/err1', function (req, res, next) {
  next(new Error('boom'))
})
app.post('/err2', function (req, res) {
  throw new Error('boom thrown')
})

then the APM instrumentation for Express will (a) capture a transaction (like it does for any incoming request) and (b) a separate "error" object. When captureBody: "errors" is configured, then it is only on the latter that the body is captured.

Here is a small Express app that shows it working.

// capturebody.example.js

const apm = require('elastic-apm-node').start({
  // serverUrl: '...',
  // secretToken: '...',
  serviceName: 'capturebody-example',
  apiRequestTime: '2s',
  metricsInterval: '0s',
  captureBody: 'errors'
})

const bodyParser = require('body-parser')
const express = require('express')

const app = express()
app.use(bodyParser.json())
app.use(bodyParser.urlencoded({extended: false}))
app.post('/ping', function (req, res) {
  console.log(`received request: ${req.method} ${req.url}, content-type:${req.headers['content-type']}`)
  res.send({ping: 'pong'})
})
app.post('/err1', function (req, res, next) {
  console.log(`received request: ${req.method} ${req.url}, content-type:${req.headers['content-type']}`)
  next(new Error('boom sent to next'))
})
app.post('/err2', function (req, res) {
  console.log(`received request: ${req.method} ${req.url}, content-type:${req.headers['content-type']}`)
  throw new Error('boom thrown')
})
app.listen({ port: 3000 }, async () => {
  console.log('listening at <http://127.0.0.1:3000/>')
})

When I run that and call each of the endpoints:

curl -i http://127.0.0.1:3000/ping -d '{"foo":"bar"}' -H content-type:application/json
curl -i http://127.0.0.1:3000/err1 -d '{"foo":"bar"}' -H content-type:application/json
curl -i http://127.0.0.1:3000/err2 -d '{"foo":"bart"}' -H content-type:application/json

If you click on the transaction (the HTTP 5xx POST green bar here):

then that document does not include the body. However, if you click on "> View related error", then that takes you to the "error" object:

Unfortunately, I don't believe the Kibana APM app has a UI for showing a possibly-captured request body. Instead you have to click "View occurrence in Discover" -- which is the raw UI for viewing any documents stored in the underlying Elasticsearch. There you find the http.request.body.original in the document:

Improving this

  1. I think that the config var docs could be a little bit clearer here.
  2. I also wonder about changing two things in the implementation: (a) If a request body is not configured to be captured, then the http.request.body field should not be reported rather than reported as [REDACTED]. (b) I wonder if we could have the body capturing for a transaction notice if there is a reported error that is associated with this transaction; if so and if captureBody: 'errors', then we capture the body on the transaction object as well. I think that is what you would have expected, and having the body on the transaction object makes it much more accessible in the UI.

I'll open a issue for these. Done.

Hi @trentm Thanks for the reply.

I think I got my confusion here, my APIs do not throw Error instead it wrapped error message in 500 status like below.

res.status(500).send({code: 500, message: 'forecast not found', plotDetails});

so it was just showing me 500 errors on transaction but there was no separate Error Object like you have shown.

Although just to double check, I have configured same way as in example you have given and I am still not getting request body.
Code:

const apm = require('elastic-apm-node').start({
  // Override the service name from package.json
  // Allowed characters: a-z, A-Z, 0-9, -, _, and space
  serviceName: 'forecast-service',

  // Use if APM Server requires a secret token
  secretToken: 'token',

  // Set the custom APM Server URL (default: http://localhost:8200)
  serverUrl: 'url',
  ignoreUrls: ['/', '/metrics'],
  captureBody: 'errors'
});

const app = express();
// logger
app.use(expressLogger);
app.use(express.json());
app.use(express.text());
app.use(express.urlencoded({extended: false}));

app.get('/', (req, res) => {
  res.status(200).send();
});

app.get('/metrics', async (req, res) => {
  // Start the timer
  const end = httpRequestDurationMicroseconds.startTimer();
  const route = req.route.path;

  res.setHeader('Content-Type', register.contentType);
  res.send(await register.metrics());

  // End timer and add labels
  end({route, code: res.statusCode, method: req.method});
});

app.post('/err1', async (req, res, next) => {
  console.log(`received request: ${req.method} ${req.url}, content-type:${req.headers['content-type']}`);
  next(new Error('boom sent to next'));
});
  1. I am not able to see transaction for failed requests. (as shown in your example)
  2. Error sample exist as shown in below screenshot.
  3. When I open "View Occurrence in Discover', its not showing me Request Body
    (Since I am new user I cant attach multiple Images, so just to show I have merged 3 images in single one. I hope you get the idea :slight_smile: )

Let me know if I am missing anything here.
As of now is there anyway I can capture HTTP Body if I get response code != 200 or response code == 500.

Oh, I didn't know about that restriction. That's a pain. Thanks for putting multiple screenshots together.

Hrm. I'm not sure why that is. If you want to try to debug this, you can add something like this:

apm.addErrorFilter(error => {
  console.log('About to send this APM error:')
  console.dir(error, { depth: 5 })
  console.log('Did we capture the body?', error.context.request.body)
  return error
})

That is using the addErrorFilter API that allows changing the error object that the APM agent is about to report. In this case we just use it to print out what that error object includes.

I'm curious if that shows that the APM agent has captured an error body when you POST /err1.

Yes, the APM instrumentation of Express is only capturing an "error" in the sense of Express error handling as described at Express error handling

I think this could be managed with addTransactionFilter.

const apm = require('elastic-apm-node').start({
  // ...
  captureBody: 'transactions'
})

apm.addTransactionFilter(trans => {
  // console.log('trans:'); console.dir(trans, { depth: 5 })

  // Remove the captured body if status code is <500.
  if (trans?.context?.response?.status_code < 500 && trans.context?.request?.body) {
    delete trans.context.request.body
  }
  return trans
})

// ...

I think that should work. This tells the agent to capture the request body for all transactions, and then we drop that captured body after the fact based on the response status code.

Thanks @trentm for the reply. Adding transaction filter has solved my issue and I am able to capture body for all my failed http transactions.

I have tried debugging more on error body just for curiosity but its not even going to addErrorFilter in my case in the case of

app.post('/err1', async (req, res, next) => {
  console.log(`received request: ${req.method} ${req.url}, content-type:${req.headers['content-type']}`);
  next(new Error('boom sent to next'));
});

Since its not really my use case any more I have not spent much time debugging this.

After resolving my request body issue with your help, I am trying to configure aws lambda serverless functions to APM and I was not able to find a way to add event.json to my transactions as well :smiley:

Since it was not related to our current discussion, I have created new discussion here

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.