# AWS S3 getObject failure: "signature mismatch" while updating APM to 2.x **URL:** \ **Category:** APM\ **Created:** [March 21, 2019, 9:23am UTC](https://discuss.elastic.co/t/aws-s3-getobject-failure-signature-mismatch-while-updating-apm-to-2-x/173275 "2019-03-21T09:23:14Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Delapouite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/delapouite/32/42489_2.png) [@Delapouite](https://discuss.elastic.co/u/Delapouite)\ **Post date:** [March 21, 2019, 9:23am UTC](https://discuss.elastic.co/t/aws-s3-getobject-failure-signature-mismatch-while-updating-apm-to-2-x/173275/1 "2019-03-21T09:23:15Z")
Hi I work on a Node.js app using the official aws-sdk for fetching stuffs from an S3 bucket: [GitHub - aws/aws-sdk-js: AWS SDK for JavaScript in the browser and Node.js](https://github.com/aws/aws-sdk-js) Relevant part in pseudo-code: ``` startElasticApm() const s3 = new S3(credentials) const { Body } = await s3.getObject({ Bucket: 'foo', Key: 'bar' }).promise() // deal with the Body… ``` This code works great with apm-agent-nodejs 1.x Now, when I attempt to update apm-agent-nodejs to 2.x, calls to S3 fail with the following errors: > The request signature we calculated does not match the signature you provided. Check your key and signing method. Code is the same, credentials are the same (and still valid). I have no clue on how the S3 lib computes the signature internally, but it seems that the apm instrumentation mangles with the request and therefore temper with the signing process. By git bisecting commits, I think I found the guilty one: [feat: add support for Distributed Tracing (#538) · elastic/apm-agent-nodejs@afd8bac · GitHub](https://github.com/elastic/apm-agent-nodejs/commit/afd8bac827554d7cf11412036ef50a3e4d1eb891) Before this commit adding support for Distributed tracing, there was no S3 signature problem. Thanks. ---
**Author:** ![wa7son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wa7son/32/21259_2.png) [@wa7son](https://discuss.elastic.co/u/wa7son)\ **Post date:** [March 21, 2019, 10:04am UTC](https://discuss.elastic.co/t/aws-s3-getobject-failure-signature-mismatch-while-updating-apm-to-2-x/173275/2 "2019-03-21T10:04:41Z")
Hi @Delapouite, welcome to the Discuss forum and thanks for raising this issue. To support distributed tracing, the agent will automatically add a new HTTP header called `elastic-apm-traceparent` to all outgoing HTTP requests. My best guess is that this happens after the signature is calculated and that the request therefore fails the signature test. I'll need to first reproduce this to see if this is in fact what's going on, and then see what can be done to fix it. But unfortunately there's no way around it currently. ---
**Author:** ![illuday](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/illuday/32/42819_2.png) [@illuday](https://discuss.elastic.co/u/illuday)\ **Post date:** [March 26, 2019, 7:08pm UTC](https://discuss.elastic.co/t/aws-s3-getobject-failure-signature-mismatch-while-updating-apm-to-2-x/173275/3 "2019-03-26T19:08:42Z")
I've the exact same issue with S3 (upload). Can we downgrade - elastic-apm-node - to make it work ? ---
**Author:** ![wa7son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wa7son/32/21259_2.png) [@wa7son](https://discuss.elastic.co/u/wa7son)\ **Post date:** [March 27, 2019, 3:21pm UTC](https://discuss.elastic.co/t/aws-s3-getobject-failure-signature-mismatch-while-updating-apm-to-2-x/173275/4 "2019-03-27T15:21:21Z")
I've made a PR to make sure we never add the `elastic-apm-traceparent` header to S3 requests: [https://github.com/elastic/apm-agent-nodejs/pull/952](https://github.com/elastic/apm-agent-nodejs/pull/952) This is not the best solution as it only works for the domain `s3.amazonaws.com`, but until we a more general purpose solution, this should fix your particular problem. ---
**Author:** ![wa7son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wa7son/32/21259_2.png) [@wa7son](https://discuss.elastic.co/u/wa7son)\ **Post date:** [March 28, 2019, 10:13pm UTC](https://discuss.elastic.co/t/aws-s3-getobject-failure-signature-mismatch-while-updating-apm-to-2-x/173275/5 "2019-03-28T22:13:30Z")
The fix have just been released in version 2.7.1. Thanks again for letting us know about this issue 🙂 ---
**Author:** ![Delapouite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/delapouite/32/42489_2.png) [@Delapouite](https://discuss.elastic.co/u/Delapouite)\ **Post date:** [March 29, 2019, 8:49am UTC](https://discuss.elastic.co/t/aws-s3-getobject-failure-signature-mismatch-while-updating-apm-to-2-x/173275/6 "2019-03-29T08:49:12Z")
Thanks for the fix wa7son. It works! But this got me concern about this header which is now transparently sent to any "external services". It's not that sensitive of an information, but I'm afraid that my colleagues of the security team doing penetration tests will find it a bit worrisome. For now the s3 regex is hard-coded, but I'm sure that in the short term you plan to add a way to have a configurable white list of domain/host/ips to explicitly decide when to send this header. ---
**Author:** ![wa7son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wa7son/32/21259_2.png) [@wa7son](https://discuss.elastic.co/u/wa7son)\ **Post date:** [March 29, 2019, 9:57am UTC](https://discuss.elastic.co/t/aws-s3-getobject-failure-signature-mismatch-while-updating-apm-to-2-x/173275/7 "2019-03-29T09:57:57Z")
Glad to hear that 😊 > [@Delapouite](#): > > But this got me concern about this header which is now transparently sent to any "external services". It's not that sensitive of an information, but I'm afraid that my colleagues of the security team doing penetration tests will find it a bit worrisome. We did discuss this on the team previously, but didn't find any security issues with it at the time. If you could elaborate on how this might be an issue, please let us know. > [@Delapouite](#): > > For now the s3 regex is hard-coded, but I'm sure that in the short term you plan to add a way to have a configurable white list of domain/host/ips to explicitly decide when to send this header. Exactly. I wanted to get this out now to fix your issue, so instead of going through a lengthy process of standardizing this across our different languages, I just put in this quick fix for now. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 19, 2019, 5:58am UTC](https://discuss.elastic.co/t/aws-s3-getobject-failure-signature-mismatch-while-updating-apm-to-2-x/173275/8 "2019-04-19T05:58:00Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.