# Why is the stack exception captured by the apm agent different from the format in the log

**URL:** <https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662>\
**Category:** APM\
**Tags:** dotnet\
**Created:** [May 8, 2020, 5:24am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662 "2020-05-08T05:24:13Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [May 8, 2020, 5:24am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/1 "2020-05-08T05:24:13Z")

</div>

log file format

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/3/f333473bde0dffc12fd51a42e9270207506e6def.png)

apm Span stack details

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8bd647424412a8c02482a7c7a8f2efc36e827396.png)

I can't understand the stack information provided by apm.

---

<div class="post-metadata">

**Author:** ![GregKalapos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregkalapos/32/37205_2.png) [@GregKalapos](https://discuss.elastic.co/u/GregKalapos)\
**Post date:** [May 11, 2020, 4:46pm UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/2 "2020-05-11T16:46:39Z")

</div>

The screenshot you pasted from Kibana (the 2. one) is a stacktrace for a given span and not for a given error - if there is also an error associated with that span then there should be another stack trace on the error itself - but the 2 stack traces you show can't be the same - the 1. one shows an exception, the 2. doesn't.

Now, having said that - there is still room for improvement on capturing these stack traces. First of all we could do better with async methods - there is an issue for that already in the agent repo.

Also, on the 2. screenshot I see that an HTTP request happens - so that is an automatically captured span for an outgoing HTTP request. Unfortunately the way we capture the HTTP request does not give us a callstack that'd also contain the user code - we only see the stack after the async call. To improve this I opened [this issue](https://github.com/elastic/apm-agent-dotnet/issues/844).

Nevertheless for the exception I'd expect an error to show up and the callstack and that error should be very similar to what you show on your 1. screenshot.

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [May 12, 2020, 1:14am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/3 "2020-05-12T01:14:07Z")

</div>

@GregKalapos  
Thank you for your reply, sorry I did not find a picture consistent with the apm stack, so I put a similar picture.

Indeed, I need HTTP request callstack.

I want to see the cause of a back-end 500 error, apm does not show me the stack information similar to the log file, but we are used to looking at the stack format in the log.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/5/a5588456a28c3e1aff00dd1349c8f7a812a32d97.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/b/ab29f634a977533aa2d52e94f3ee5a36bc5a7b7b.png)

Finally, do I need to wait for the next version to see the "stack format in the log file"?

---

<div class="post-metadata">

**Author:** ![GregKalapos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregkalapos/32/37205_2.png) [@GregKalapos](https://discuss.elastic.co/u/GregKalapos)\
**Post date:** [May 23, 2020, 12:22pm UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/4 "2020-05-23T12:22:19Z")

</div>

Hi @wajika

I think you asked also on the GitHub PR, but let’s just also follow up here.

So, yeah, first of all there were some stack trace related PRs merged, and in the next release you’ll be able to see where in your code the outgoing HTTP request happens.

Now, I’d like to also add some comments to the screenshot you sent:

As it seems the outgoing HTTP request itself returned HTTP200, but your service (the `POST IoT/CheckAndGetProductInfo`) returned HTTP 500.

Of course I don’t know the reason for that, but I would like to mention that if you just set the return type to HTTP500 in your service but no exception leaves the pipeline then the agent won’t be able to capture the error. Similarly if you for example catch every single exception in your service and just return HTTP500 then no exception will leave the pipeline and we won’t be able capture any exception either.

In those cases the easiest is to just capture the exception manually when you handle it. [Here](https://www.elastic.co/guide/en/apm/agent/dotnet/current/public-api.html#api-transaction-capture-exception) is some doc on it.

So if you have some global error handling part, that makes sure no exception leaves the pipeline and your service just return HTTP500, you can do something like this:

```auto
catch (Exception e) // Some global error handler
{
   Agent.Tracer.CurrentTransaction?.CaptureException(e); 
   // rest of your code
}

```

That code will add the exception to your transaction and it'll show up on the UI.

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [May 25, 2020, 12:50am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/5 "2020-05-25T00:50:17Z")

</div>

@GregKalapos

Thank you for your reply.

Sorry. I don't understand what you mean

question 1

> As it seems the outgoing HTTP request itself returned HTTP200, but your service (the POST IoT/CheckAndGetProductInfo) returned HTTP 500.

In this case, does APM consider it a success?

question 2

> no exception leaves the pipeline then the agent won’t be able to capture the error  
> AND  
> Similarly if you for example catch every single exception in your service and just return HTTP500 then no exception will leave the pipeline and we won’t be able capture any exception either.

I didn't understand

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [May 26, 2020, 8:54am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/6 "2020-05-26T08:54:18Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/c/e/ce8459164ecc41428b2a4b217ef6d55b8efc3c10.png)

I still have one thing I do n’t understand, the service generated an http500 error, why is it showing the error code of apm agent?

---

<div class="post-metadata">

**Author:** ![GregKalapos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregkalapos/32/37205_2.png) [@GregKalapos](https://discuss.elastic.co/u/GregKalapos)\
**Post date:** [May 26, 2020, 9:52am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/7 "2020-05-26T09:52:08Z")

</div>

Hi @wajika

Question1:

What will happen is that the `StatusCode` will be set to HTTP500, which the UI will show with a red background, but no error will be captured.

Question2:

Let me illustrate this with some code. Let’s say you have something like this:

```auto

app.Run(async context =>
{
   context.Response.StatusCode = (int)HttpStatusCode.InternalServerError;
   await context.Response.WriteAsync("Hello, World!");
});

```

Now the `context.Response.StatusCode = (int)HttpStatusCode.InternalServerError;` could be anywhere… if you have let’s say [ASP.NET](http://ASP.NET) Core MVC and in a controller method you do something like this, then it’s the same:

```auto

public IActionResult Index()

{

   try
   {
      //Do some work
   }
   catch (Exception e)
   {
      return StatusCode(500);
   }

   return View();
}

```

No exception leaves the pipeline in those cases, so the agent has no chance to capture it for you. It’ll capture the HTTP500 which is the response code of the request, but won’t capture an error, since there was no error in the pipeline - in the 1. snippet there is no exception et al, in the 2. one you handled it. That’s why I suggested capturing the error manually in my previous comment.

On the other hand, if you do this:

```auto

app.Run(async context =>
{
   throw new Exception();
});

```

or this:

```auto

public IActionResult Index()
{
   try
   {
      //Do some work
   }
   catch (Exception e)
   {
      throw;
   }

   return View();
}

```

Then there is an exception leaving the pipeline so the agent will observe that and show an error on the APM UI.

> I still have one thing I do n’t understand, the service generated an http500 error, why is it showing the error code of apm agent?

Sorry, I don’t fully understand. If your question is why you see APM code on the callstack then it’s because the agent subscribes to some internal events therefore at the point of the stack trace capturing the agent is already on the callstack. We don’t trim those from the callstack - we show you the real callstack and since there are agent frames on the callstack those just show up.

If your question is different feel free to elaborate.

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [June 4, 2020, 3:20am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/8 "2020-06-04T03:20:00Z")

</div>

@GregKalapos  
Let me talk about my own thoughts.

For example, if a http500 error occurs, it belongs to the backend service error, then I need to find out which line of code of the backend service is causing the problem from the apm stack information. (But I didn't find where the stack of the service is displayed)

Similar to the picture below.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/6/e66c660ce0c792c9e6620457e88a7174970fd02a.jpeg)

But after using apm, I found that the information generated by elastic apm is different from my needs, so I don’t know if elastic apm does not have this function or my usage is wrong.

I am not a developer. I communicated with the developers. Our project uses a unified interceptor to intercept errors.

The code looks like this:  
[https://paste.ubuntu.com/p/ZxCHD8KjpJ/](https://paste.ubuntu.com/p/ZxCHD8KjpJ/)

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [June 8, 2020, 8:40am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/9 "2020-06-08T08:40:03Z")

</div>

@GregKalapos  
I think error tracking and apm should be closely related. Has the elastic team considered this when designing the product?

---

<div class="post-metadata">

**Author:** ![GregKalapos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregkalapos/32/37205_2.png) [@GregKalapos](https://discuss.elastic.co/u/GregKalapos)\
**Post date:** [June 8, 2020, 9:42am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/10 "2020-06-08T09:42:59Z")

</div>

Thanks for the code snippet @wajika, that makes the situation clear.

So you have exactly the situation I described earlier - you have code in the application which handles the exception, therefore the agent has no way to detect it. In this case you have an [ASP.NET](http://ASP.NET) Core filter which handles the exception and sets the status code manually.

This happens in line 78 in the code snippet:

```auto
 context.Exception = null; //Handled!

```

With this, no exception is leaves the pipeline - there is no error to catch. Also, in the `GetStatusCode` you set the status code manually. Same as my example above. So please keep in mind that returning HTTP5xx does not mean there is an error to capture.

Now, to the solution: like I said before, you can still manually capture these exceptions before you handle them, you can do something like this:

```auto
// This here is a dummy simple exception filter to show how to capture the exception with the Elastic .NET Agent API
public class SampleExceptionFilter: IExceptionFilter
{
	public void OnException(ExceptionContext context)
	{
		Elastic.Apm.Agent.Tracer?.CurrentTransaction.CaptureException(context.Exception);
		context.ExceptionHandled = true;
	}
}

```

If you rely this to .NET developers, I think they will be able to understand and take action on this.

To you question:

> I think error tracking and apm should be closely related. Has the elastic team considered this when designing the product?

Those are closely related - we capture errors on spans and transactions and you can jump from the errors to transactions and vice versa.

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [June 9, 2020, 1:49am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/11 "2020-06-09T01:49:32Z")

</div>

We succeeded, it was really great. thank you very much for your help.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/5/1562214069bc27fcd2efc859535cb48bc71185a9.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/c/9ca3aaf024b5d1629a994cf419fe3cb0855c16d8.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/1/1176f05959f08043f887497b1e5ce02d7af60753.png)

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [June 9, 2020, 6:21am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/12 "2020-06-09T06:21:36Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/3/0/30c349db897c309210600ec0a1a239b271534580.png)

In addition, I would like to make another suggestion. Can the Transaction column support time ordering?

We feel inconvenient when looking for the latest record.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 30, 2020, 2:21am UTC](https://discuss.elastic.co/t/why-is-the-stack-exception-captured-by-the-apm-agent-different-from-the-format-in-the-log/231662/13 "2020-06-30T02:21:42Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.