AWS S3 getObject failure: "signature mismatch" while updating APM to 2.x

Elastic Observability APM
1

Hi

I work on a Node.js app using the official aws-sdk for fetching stuffs from an S3 bucket: GitHub - aws/aws-sdk-js: AWS SDK for JavaScript in the browser and Node.js

Relevant part in pseudo-code:

startElasticApm()
const s3 = new S3(credentials)
const { Body } = await s3.getObject({ Bucket: 'foo', Key: 'bar' }).promise()
// deal with the Body…

This code works great with apm-agent-nodejs 1.x

Now, when I attempt to update apm-agent-nodejs to 2.x, calls to S3 fail with the following errors:

The request signature we calculated does not match the signature you provided. Check your key and signing method.

Code is the same, credentials are the same (and still valid). I have no clue on how the S3 lib computes the signature internally, but it seems that the apm instrumentation mangles with the request and therefore temper with the signing process.

By git bisecting commits, I think I found the guilty one: feat: add support for Distributed Tracing (#538) · elastic/apm-agent-nodejs@afd8bac · GitHub

Before this commit adding support for Distributed tracing, there was no S3 signature problem.

Thanks.

1 Like
2

Hi @Delapouite, welcome to the Discuss forum and thanks for raising this issue.

To support distributed tracing, the agent will automatically add a new HTTP header called elastic-apm-traceparent to all outgoing HTTP requests. My best guess is that this happens after the signature is calculated and that the request therefore fails the signature test.

I'll need to first reproduce this to see if this is in fact what's going on, and then see what can be done to fix it. But unfortunately there's no way around it currently.

3

I've the exact same issue with S3 (upload).

Can we downgrade - elastic-apm-node - to make it work ?

4

I've made a PR to make sure we never add the elastic-apm-traceparent header to S3 requests: https://github.com/elastic/apm-agent-nodejs/pull/952

This is not the best solution as it only works for the domain s3.amazonaws.com, but until we a more general purpose solution, this should fix your particular problem.

5

The fix have just been released in version 2.7.1.

Thanks again for letting us know about this issue :slight_smile:

6

Thanks for the fix wa7son. It works!

But this got me concern about this header which is now transparently sent to any "external services". It's not that sensitive of an information, but I'm afraid that my colleagues of the security team doing penetration tests will find it a bit worrisome.

For now the s3 regex is hard-coded, but I'm sure that in the short term you plan to add a way to have a configurable white list of domain/host/ips to explicitly decide when to send this header.

7

Glad to hear that :blush:

We did discuss this on the team previously, but didn't find any security issues with it at the time. If you could elaborate on how this might be an issue, please let us know.

Exactly. I wanted to get this out now to fix your issue, so instead of going through a lengthy process of standardizing this across our different languages, I just put in this quick fix for now.

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.