# Only common trace ID for sampled spans

**URL:** <https://discuss.elastic.co/t/only-common-trace-id-for-sampled-spans/192459>\
**Category:** APM\
**Tags:** go\
**Created:** [July 26, 2019, 2:22pm UTC](https://discuss.elastic.co/t/only-common-trace-id-for-sampled-spans/192459 "2019-07-26T14:22:31Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ricco\_Forgaard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ricco_forgaard/32/51019_2.png) [@Ricco\_Forgaard](https://discuss.elastic.co/u/Ricco_Forgaard)\
**Post date:** [July 26, 2019, 2:22pm UTC](https://discuss.elastic.co/t/only-common-trace-id-for-sampled-spans/192459/1 "2019-07-26T14:22:31Z")

</div>

Hi,

We have a bunch of micro services implemented in Go that talk to each other over gRPC.

Recently we implemented logging of the trace ID in our services in order to be able to trace logs across services. We use the Zap logger and extract the APM information to create a new logger as described [here](https://www.elastic.co/guide/en/apm/agent/go/current/instrumenting-source.html#builtin-modules-apmzap).

However, it seems like the trace ID only gets sent from one service to the next if the current span is sampled. If the span isn't sampled we get a (seemingly) random trace ID and we're unable to trace the request across services.

We instrument our clients with the interceptor like this:

```auto
conn, err := grpc.Dial(addr, grpc.WithInsecure(), grpc.WithUnaryInterceptor(apmgrpc.NewUnaryClientInterceptor()))

```

Looking at the [interceptor's source code](https://github.com/elastic/apm-agent-go/blob/v1.4.0/module/apmgrpc/client.go#L56) we can see that the metadata doesn't get added if the span is dropped:

```auto
func startSpan(ctx context.Context, name string) (*apm.Span, context.Context) {
	span, ctx := apm.StartSpan(ctx, name, "external.grpc")
	if span.Dropped() {
		return span, ctx
	}
	traceparentValue := apmhttp.FormatTraceparentHeader(span.TraceContext())
	md, ok := metadata.FromOutgoingContext(ctx)
	if !ok {
		md = metadata.Pairs(traceparentHeader, traceparentValue)
	} else {
		md = md.Copy()
		md.Set(traceparentHeader, traceparentValue)
	}
	return span, metadata.NewOutgoingContext(ctx, md)
}

```

This means that for a sample rate lower than 1.0 tracing doesn't really work as intended.

Now the questions are:

- are we doing anything wrong?
- have we misunderstood how distributed tracing works?
- is this a bug in the module?

We're using version 1.4.0 of the APM gRPC module.

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [July 29, 2019, 2:37am UTC](https://discuss.elastic.co/t/only-common-trace-id-for-sampled-spans/192459/2 "2019-07-29T02:37:17Z")

</div>

Hi @Ricco_Forgaard,

Welcome to the forum, and thanks for raising your question here. You have indeed found a bug in the `apmgrpc` module. The client interceptor should behave similarly to the `apmhttp` client instrumentation, including the transaction's trace context if the span is dropped.

I have opened [https://github.com/elastic/apm-agent-go/issues/601](https://github.com/elastic/apm-agent-go/issues/601) to get this fixed.

---

<div class="post-metadata">

**Author:** ![Ricco\_Forgaard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ricco_forgaard/32/51019_2.png) [@Ricco\_Forgaard](https://discuss.elastic.co/u/Ricco_Forgaard)\
**Post date:** [July 29, 2019, 6:31am UTC](https://discuss.elastic.co/t/only-common-trace-id-for-sampled-spans/192459/3 "2019-07-29T06:31:32Z")

</div>

Thanks @axw, good to know.

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [July 31, 2019, 11:49pm UTC](https://discuss.elastic.co/t/only-common-trace-id-for-sampled-spans/192459/4 "2019-07-31T23:49:15Z")

</div>

In case you aren't watching the issue: we've just released [v1.5.0](https://github.com/elastic/apm-agent-go/releases/tag/v1.5.0) of the Go agent which includes a fix for this. Thanks again for bringing it to our attention.

---

<div class="post-metadata">

**Author:** ![Ricco\_Forgaard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ricco_forgaard/32/51019_2.png) [@Ricco\_Forgaard](https://discuss.elastic.co/u/Ricco_Forgaard)\
**Post date:** [August 1, 2019, 10:17am UTC](https://discuss.elastic.co/t/only-common-trace-id-for-sampled-spans/192459/5 "2019-08-01T10:17:49Z")

</div>

I did, thanks for responding so quickly! Highly appreciated.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 22, 2019, 6:17am UTC](https://discuss.elastic.co/t/only-common-trace-id-for-sampled-spans/192459/6 "2019-08-22T06:17:49Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.