Duplicate transactions count in elastic apm

Elastic Observability APM
1

Kibana version: 7.6.2

Elasticsearch version:7.6.1

APM Server version:7.6.1

APM Agent language and version:1.16.0, 1.17.0, java-agent

Browser version: not relevant

Original install method (e.g. download page, yum, deb, from source, etc.) and version:yum

Fresh install or upgraded from other version?:fresh

Is there anything special in your setup? nothing, separate indexes for transactions, spans, metrics, errors etc made on apm server, also we drop unsampled transactions via processors on apm server to lower throughput

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

We are having a weird issue with transaction count. We have several services (tomcat apps mostly) which connect to each other via various endpoints. Somehow we noticed that spans from transaction on service A affect transactions count on service B. If we change sample rate for service A 0.01>0.001 we will see the drop of transactions for service A, but also we will see the drop for service B.
Also in some cases the drop for the service is not equal to sample rate setting change (example we change frame rate 1 > 0.1 and see 40000tpm > 25000tpm change which is not relevant)

We believe that this happens because transactions from "related" services are somehow summed up.

Please clarify this situation, because right now we cannot know for sure how much tpm exactly we get for each app.

Steps to reproduce:

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

2

Hello Alexey,

The elastic agent transfers 2 headers when calling another service:

  • traceparent
  • tracestate

The traceparent contains the TransactionId of the current service so both transactions can be connected.
The tracestate is set depending on the fact if the current transaction has been traced. If this is not the case all following services will not trace.

Best regards
Wolfram

PS: It might also be the sampled flag in the traceparent header - I am not sure

2 Likes
3

where can I read more about this?

4

The names of the headers can be found here.
The description of the headers can be found in this W3C document.

5

Can you explain this a bit? I don`t get how this can affect tpm count in my situation?

6

I was just explaining why you see a drop in transactions for service B if you reduce the sample rate for service A.

7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.