Span.End() method seems to lock the application in certain cases

Elastic Observability APM
1

We recently implemented our apm Custom spans in our code (that processes kafka messages in a high speed) and thanks to one of our processes timeouts and further debug logging we catch that a custom Span End method was causing the program to freeze in that line of code.
We tried with the defer span.End() and as a normal function called and the application froze regardless. After removing the span, we did not see the problem anymore, confirming it was there the problem.
We have set up a 60 s timeout in our parent transaction timeout, but does nothing in relation to that issue.
Any idea what may cause this?
APM Agent Golang version v1.11.0

2

@javiersotodev welcome to the forum! Sorry to hear you're experiencing this issue.

This is certainly not expected, and we haven't had any similar reports. We'll need some more information to debug.

Can you please share a stack trace dump? If you're not familiar with how to do this, Debugging Go with stack traces (evanjones.ca) has some tips.

1 Like
3

It has been hard to reproduce in our lower environments as it happened in production (probably due to a high volume) While i implement the debug stack trace and replicating the incident:

  • What is it the better way to implement custom spans, implementing them from the transaction or the parent's transaction context? Because it doesnt throw any errors when it freezes, we are mostly blind in this.
  • We send our contexts into go routines to track spans inside the main transaction, with their respectives end() methods in defer.
    Any Help or advice would be appreciated while i get the trace stack for you
    Thank you
4

Are you asking if it's better to use Transaction.StartSpan(...) vs. apm.StartSpan(ctx, ...)? If so, neither is generally better than the other. apm.StartSpan will internally call Transaction.StartSpan. I would say the context approach is typically most appropriate though.

We send our contexts into go routines to track spans inside the main transaction, with their respectives end() methods in defer.

This should be fine.

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.