Is it possible to limit transaction sampling rate by transaction duration?

[Nate_M (Nate Moseman)]

We are using Elastic APM with Nodejs right now. I have a transactionSampleRate set to 0.2 to lower the overhead of the sampling on the APM servers and Elasticsearch storage.

This works very well for the most part. However sometimes when the website gets busy we will see a small number of very slow transactions for one reason or another and it is difficult to troubleshoot when those are the transactions were the sampling is skipped.

So we are looking for a solution were 100% of slow transactions are sampled and sent to APM server, but only a 20% of normal transactions are.

Is it possible to do something like this?

I thought maybe something like addSampleFilter() might exist or maybe just use addFilter() to creatively edit the sampled transactions. But after messing with it for a while it didn't seem like a fruitful approach.

I would be happy to ignore 100% of the normal transactions if that helps.

[axw (Andrew Wilkins)]

@Nate_M welcome to the forum!

This is not currently possible in general. We are working on a tail-based sampling feature to do something similar to what you're describing.

Our current plan is to do something like this:

• configure N% of traces to capture, based on the service and optionally root transaction group
• APM Agent sends all traces to the APM Server
• APM Server samples at most N% traces per group, preferring to keep the slower ones

In the situation where there are relatively few slow transactions, we would have a greater chance of sampling them.

[Nate_M (Nate Moseman)]

That makes sense. Thank you.

Any ETA or roadmap for this tail-based sampling feature?

I am curious since it's a feature we would value highly.

[axw (Andrew Wilkins)]

No ETA available at the moment. We've started work on it, but there's still a substantial amount of work remaining. I recommend subscribing to the issue for updates.