APM, Jaeger and trace.id

[tstadler (Tobias Stadler)]

Kibana version: 7.7.0

Elasticsearch version: 7.7.0

APM Server version: 7.7.0

I am using the apm-server with jaeger grpc enabled and a jaeger tracer as client and ECS logging in my application.

The apm-server stores the first span with the fields transaction.id and trace.id. The value of transaction.id is e.g. 7890b687652000e, while the value of trace.id is 0000000000000000a7890b687652000e. The value of the trace.id field in my application logs is 7890b687652000e.

When using the trace logs action in the APM app, the search field in populated with trace.id:"0000000000000000a7890b687652000e" OR 0000000000000000a7890b687652000eand no logs are found. When I change the query to trace.id:"a7890b687652000e" OR a7890b687652000e my logs are found.

Is it possible to store the trace.id field without the leading zeros?

[axw (Andrew Wilkins)]

It sounds to me like the Jaeger client you're using is generating 64-bit trace IDs. I think it would be reasonable for APM Server to detect 64-bit IDs in the Jaeger intake, and drop the first 16 zeros. We would still left-pad with zeros to 16 characters though.

So a7890b687652000e would be recorded as-is, but 7652000e would be recorded as 000000007652000e. I believe that would be in line with what Jaeger clients do, e.g. here's how the Jaeger Go client formats trace IDs: https://github.com/jaegertracing/jaeger-client-go/blob/v2.23.1/span_context.go#L333

How does that sound to you?

[tstadler (Tobias Stadler)]

Sounds great. Thank you!

[axw (Andrew Wilkins)]

Great. I've filed an issue, please subscribe if you would like to be updated: https://github.com/elastic/apm-server/issues/3837.