AWS S3 getObject failure: "signature mismatch" while updating APM to 2.x

[Delapouite (Bruno Heridet)]

Hi

I work on a Node.js app using the official aws-sdk for fetching stuffs from an S3 bucket: GitHub - aws/aws-sdk-js: AWS SDK for JavaScript in the browser and Node.js

Relevant part in pseudo-code:

startElasticApm()
const s3 = new S3(credentials)
const { Body } = await s3.getObject({ Bucket: 'foo', Key: 'bar' }).promise()
// deal with the Body…

This code works great with apm-agent-nodejs 1.x

Now, when I attempt to update apm-agent-nodejs to 2.x, calls to S3 fail with the following errors:

The request signature we calculated does not match the signature you provided. Check your key and signing method.

Code is the same, credentials are the same (and still valid). I have no clue on how the S3 lib computes the signature internally, but it seems that the apm instrumentation mangles with the request and therefore temper with the signing process.

By git bisecting commits, I think I found the guilty one: feat: add support for Distributed Tracing (#538) · elastic/apm-agent-nodejs@afd8bac · GitHub

Before this commit adding support for Distributed tracing, there was no S3 signature problem.

Thanks.

[wa7son (Thomas Watson)]

Hi @Delapouite, welcome to the Discuss forum and thanks for raising this issue.

To support distributed tracing, the agent will automatically add a new HTTP header called elastic-apm-traceparent to all outgoing HTTP requests. My best guess is that this happens after the signature is calculated and that the request therefore fails the signature test.

I'll need to first reproduce this to see if this is in fact what's going on, and then see what can be done to fix it. But unfortunately there's no way around it currently.

[illuday (Emmanuel Russeil)]

I've the exact same issue with S3 (upload).

Can we downgrade - elastic-apm-node - to make it work ?

[wa7son (Thomas Watson)]

I've made a PR to make sure we never add the elastic-apm-traceparent header to S3 requests: https://github.com/elastic/apm-agent-nodejs/pull/952

This is not the best solution as it only works for the domain s3.amazonaws.com, but until we a more general purpose solution, this should fix your particular problem.

[wa7son (Thomas Watson)]

The fix have just been released in version 2.7.1.

Thanks again for letting us know about this issue

[Delapouite (Bruno Heridet)]

Thanks for the fix wa7son. It works!

But this got me concern about this header which is now transparently sent to any "external services". It's not that sensitive of an information, but I'm afraid that my colleagues of the security team doing penetration tests will find it a bit worrisome.

For now the s3 regex is hard-coded, but I'm sure that in the short term you plan to add a way to have a configurable white list of domain/host/ips to explicitly decide when to send this header.

[wa7son (Thomas Watson)]

Glad to hear that

@Delapouite

But this got me concern about this header which is now transparently sent to any "external services". It's not that sensitive of an information, but I'm afraid that my colleagues of the security team doing penetration tests will find it a bit worrisome.

We did discuss this on the team previously, but didn't find any security issues with it at the time. If you could elaborate on how this might be an issue, please let us know.

@Delapouite

For now the s3 regex is hard-coded, but I'm sure that in the short term you plan to add a way to have a configurable white list of domain/host/ips to explicitly decide when to send this header.

Exactly. I wanted to get this out now to fix your issue, so instead of going through a lengthy process of standardizing this across our different languages, I just put in this quick fix for now.