Kibana version: 7.17
Elasticsearch version: 7.17
APM Server version: 7.17
APM Agent language and version: Java | 1.32.0
Original install method (e.g. download page, yum, deb, from source, etc.) and version: Everything is containerized running on K8s.
Fresh install or upgraded from other version? Upgraded from an older version.
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
For a particular service, traces for a few particular transactions are not getting sampled at all, and hence, the trace timeline is not available for those transactions.
I've already verified that the sampling rate for the service is 100%. And other transactions for the service are getting successfully sampled and hence, I've the trace timeline available for those.
I've noticed that the transactions that aren't getting sampled have below things in common:
event.outcome is unknown
http.response.finished is - (Null)
Whereas for the transactions for which traces are getting sampled, event.outcome is either success or failure and http.response.finished is true.
Yes, it is a distributed tracing system and other services are also instrumented via Elastic APM Agent(Same version, same language) and those also have sampling 100%.
What's weird is that for a particular service itself, traces for few transactions are not getting sampled at all whereas other transactions are working fine.
Provide logs and/or server output (if relevant):
Let me know what all information would be required?