How do I setup log correlation when using extension layers?

Elastic Observability APM
,
1

Hi! I'm using the following two extension layers:

arn:aws:lambda:us-west-2:267093732750:layer:elastic-apm-extension-ver-1-0-2-x86_64:1, arn:aws:lambda:us-west-2:267093732750:layer:elastic-apm-node-ver-3-38-0:1

The APM is working. Now I'm interested in setting up log correlation. But I'm not sure to get the trace and transaction IDs so I can include them in my logs. Thanks for the help!

2

Hi @wjensen-godaddy what logging library are you using? Our documentation on how to set up log correlation is here. You can inject the IDs into the logs using the apm.currentTraceIds method. Let me know if that helps or if you need more info.

3

This is what I've tried in my handler:

import apm from 'elastic-apm-node';

console.log({ currentTraceIds: apm.currentTraceIds });

And this is what I see in my logs:

INFO	{ currentTraceIds: Ids {} }

It doesn't look like it can access the trace Ids from the extension layer?

4

@wjensen-godaddy Hi. Is your console.log(...) inside the handler function? It will need to be for there to be an active transaction. For example, here is a complete handler file of mine:

const apm = require('elastic-apm-node')

exports.handler = async function myPlayHandler (event, context) {
  console.log('currentTraceIds', apm.currentTraceIds)
  return {
    statusCode: 200,
    body: 'hi from myPlayHandler'
  }
}

Invoking that Lambda function results in this line in the CloudWatch log output:

2022-12-02T19:40:53.405000+00:00 2022/12/02/[$LATEST]b42db6975b494355841ad3a4d7ba8ffc 2022-12-02T19:40:53.405Z	55622ea1-6a89-44f1-bac1-8301a2969dc9	INFO	currentTraceIds TransactionIds {
  'trace.id': 'f9ea0cef0435c7baa76c9eddac7ce9d1',
  'transaction.id': '291a35665e5c34bc'
}
5

@wjensen-godaddy Also, if you are using pino or winston, we have formatter modules that you can use with them to (a) get ecs-logging formatted output and (b) get automatic log correlation: Introduction | ECS Logging Node.js Reference | Elastic

6

I see. So this should be working. I don't know why it's not working for me. We both seem to be doing the same thing.

7

@wjensen-godaddy
Could it have to do with your usage of import ...? Is this TypeScript code that is being transpiled to JS code that uses commonjs (i.e. require) usage?

Or if this is using import (ES Modules), then there are various limitations there for the APM agent. I don't know how that would affect this case, however. Can you show a more complete example of your handler code?

8

I'm afraid our handler code is too big to share.

Ya, we're transpiling using commonjs modules.

9

I suspect esbuild might be including 'elastic-apm-node' in the bundle. I have it as a dev dependency but I forgot to mark it as external. Testing that now...

10

Ugh, that was it. :expressionless:

Thanks for the help. It might be nice if the documentation had a section on this. But maybe the solution is obvious to other people. Haha.

11

By "solution", I just mean that it's possible to do log correlation by importing the APM from the extension layer, and calling apm.currentTraceIds.

12

Ah, yes, esbuild (or the same impact for any bundler). We have some docs on this starting here: Starting the agent | APM Node.js Agent Reference [3.x] | Elastic
But that wouldn't have been obvious for you to find at all.

I really should get a pointer to this info in the troubleshooting guide: Troubleshooting | APM Node.js Agent Reference [3.x] | Elastic
I've opened add section about bundlers gotchas to the Troubleshooting docs · Issue #3055 · elastic/apm-agent-nodejs · GitHub to do that eventually.

1 Like
13

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.