We are using Elastic APM with Nodejs right now. I have a transactionSampleRate set to 0.2 to lower the overhead of the sampling on the APM servers and Elasticsearch storage.
This works very well for the most part. However sometimes when the website gets busy we will see a small number of very slow transactions for one reason or another and it is difficult to troubleshoot when those are the transactions were the sampling is skipped.
So we are looking for a solution were 100% of slow transactions are sampled and sent to APM server, but only a 20% of normal transactions are.
Is it possible to do something like this?
I thought maybe something like addSampleFilter() might exist or maybe just use addFilter() to creatively edit the sampled transactions. But after messing with it for a while it didn't seem like a fruitful approach.
I would be happy to ignore 100% of the normal transactions if that helps.