We are trying to use Elastic APM for performance monitoring and tracing. The official document said,
By default, the agent will sample every transaction (e.g. request to your service). To reduce overhead and storage requirements, you can set the sample rate to a value between 0.0 and 1.0. We still record overall time and the result for unsampled transactions, but no context information, tags, or spans.
We have tested it on our production, based on my rough estimation of our case, if all the unsampled transactions will be recorded, the total QPS is more than 100k, and the data volume in ES is also too expensive.
I'd like to ask if we can just record the things only if it is sampled, not just the spans/tags, but also the root transactions. I understood the cons is we can directly get the number like tpm via Kibana, but most of the features worked well like latency calculation and tracing. But the benefits are so big, we can save the cost and resources. I didn't find the similar configuration from the APM documentation, can anyone help?
Kibana version: 7.1
Elasticsearch version: 7.1
APM Server version: 7.1
APM Agent language and version: Python/Java/Go