How to ignore certain errors in APM?

[benjismith]

I have a nodejs/express REST API service using the Elastic APM client, and the Elastic Cloud backend, and I'm very happy with it so far.

My application regularly needs to check S3 for the existence/non-existence of certain keys in certain buckets, using the AWS S3.HeadObject method. When those keys don't exist, the S3 API returns a 404 status, which is perfectly normal behavior within my application.

But the Elastic APM client reports these as Errors, which clutters up our APM dashboard and makes it harder to implement a good alerting strategy. Is there any way we can set these errors to be ignored? They're not being directly captured in any of my application code, but instead in the instrumented S3 library.

What would you recommend for this situation?

[trentm (Trent Mick)]

Hi @benjismith. Thanks for asking!

I think the easiest would be to use the apm.addErrorFilter(...) method of the API. Perhaps something like this

apm.addErrorFilter(error => {
  if (error.exception && error.exception.code === 'NoSuchKey') {
    return null
  }
  return error
})

The "error" object returned by the current "aws-sdk" instrumentation looks like:

{
    "id": "d9b1268e25f1ad18d972bc2bb31683d9",
    "timestamp": 1653949020384000,
    "parent_id": "b1a14c1bba13ab45",
    "trace_id": "d6da51691be6ff1d4e222c211c29e522",
    "transaction_id": "0a3b3bbff338e748",
    "transaction": {
        "name": "manual",
        "type": null,
        "sampled": true
    },
    "context": {
        "user": {},
        "tags": {},
        "custom": {}
    },
    "exception": {
        "message": "The specified key does not exist.",
        "type": "NoSuchKey",
        "handled": true,
        "code": "NoSuchKey",
        "attributes": {
            "message": "The specified key does not exist.",
            "time": "2022-05-30T22:17:00.384Z",
            "requestId": "HMRXQQ867R86CQ84",
            "extendedRequestId": "Huo8ClMk7XJsKEJ06kHnbD7S/AX5D67FWxu+x3gbiE9EZr6i77gvh13iyKyoKs3LYTAkyRqKX2I=",
            "statusCode": 404,
            "retryable": false,
            "retryDelay": 69.105352438465
        },
        "module": "aws-sdk",
        "stacktrace": [
            {
                "filename": "/Users/trentm/el/apm-agent-nodejs/node_modules/aws-sdk/lib/services/s3.js",
                "lineno": 711,
                "function": "extractError",
                "library_frame": true,
                "abs_path": "/Users/trentm/el/apm-agent-nodejs/node_modules/aws-sdk/lib/services/s3.js",
                "pre_context": [
                    "      }",
                    ""
                ],
                "context_line": "      resp.error = AWS.util.error(new Error(), {",
                "post_context": [
                    "        code: data.Code || code,",
                    "        message: data.Message || null,"
                ]
            },
            ...
        ]
    },
    "culprit": "extractError (/Users/trentm/el/apm-agent-nodejs/node_modules/aws-sdk/lib/services/s3.js)"
}

One could also use that captured .attributes.statusCode .

This will tell the agent to drop any errors it would have reported with that aws-sdk error code. Does that work for you? Currently the error object doesn't tell you whether the NoSuchKey was from a HeadObject or GetObject or some other S3 API call -- other than indirectly via that parent_id, which is to the span for the API call.

[benjismith]

This is perfect. The error filter is exactly what I was looking for... Thank you!