APM .net agent see the Service Name in the logs for correlated transactions

Nicolas_Rey · August 1, 2022, 3:39pm

Kibana version:
7.16.3 (Docker)

Elasticsearch version:
7.16.3 (Docker)

APM Server version:
7.16.3 (Docker)

APM Agent language and version:
.net

    <PackageReference Include="Elastic.Apm.NetCoreAll" Version="1.16.1" />
    <PackageReference Include="Elastic.Apm.SerilogEnricher" Version="1.5.3" />
    <PackageReference Include="Elastic.Apm.StackExchange.Redis" Version="1.16.1" />

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Hi,
I use correlated transactions across multiple microservices (.net 6) using the .net APM agent, it works pretty well.
My problem is about logs, I don't know why the Service Name does not appear, so that I'm not able to distinguish which log belongs to which microservice.

Example here is an example of a correlated transaction

and the corresponding logs, but with an empty Service Name column

My Serilog logger instance is built this way :

        public static Logger GetELKLogger(IConfiguration config, string varEnv = "ElasticSearchLog")
        {
            var configuration = config.Get<Configuration>(varEnv);

            return new LoggerConfiguration()
            .ReadFrom.Configuration(config)
            .Enrich.WithElasticApmCorrelationInfo()
            .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(configuration.ElasticSearchLog))
            {
                AutoRegisterTemplate = true,
                IndexFormat = "mslogs-{0:yyyy.MM.dd}",
                DetectElasticsearchVersion = true,
                RegisterTemplateFailure = RegisterTemplateRecovery.IndexAnyway,
                AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
                FailureCallback = e => Console.WriteLine($"Unable to submit event {e?.RenderMessage()} to ElasticSearch. Exception : " + e?.Exception?.ToString()),
                EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
                                        EmitEventFailureHandling.WriteToFailureSink |
                                        EmitEventFailureHandling.RaiseCallback,
                BufferCleanPayload = (failingEvent, statuscode, exception) =>
                {
                    dynamic e = JObject.Parse(failingEvent);
                    return JsonConvert.SerializeObject(new Dictionary<string, object>()
                        {
                            { "action", "DeniedByElasticSearch"},
                            { "@timestamp",e["@timestamp"]},
                            { "level","Error"},
                            { "message","Error: "+e.message},
                            { "messageTemplate",e.messageTemplate},
                            { "failingStatusCode", statuscode},
                            { "failingException", exception}
                        });
                },
                CustomFormatter = new EcsTextFormatter()
            })
            .CreateLogger();
        }

My minimal appsettings.json for each Microservice is

  "Serilog": {
    "Using": [],
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Information",
        "Elastic": "Warning",
        "Apm": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Console"
      }
    ],
    "Enrich": [
      "FromLogContext",
      "WithMachineName",
      "WithProcessId",
      "WithThreadId"
    ],
    "Properties": {
      "ApplicationName": "IT.Microservices.AbandonedCartEmailSender"
    }
  },
  "ElasticApm": {
    "ServerUrl":"http://apm:8200",
    "Enabled": true,
    "TransactionSampleRate": 1,
    "CaptureBody": "all",
    "CaptureHeaders": true,
    "SpanFramesMinDuration": 0, // no stacktrace except for exception
    "CloudProvider": "none"
  },
  "ElasticSearchLog": {
    "ElasticSearchLog": "http://elasticsearch:9200/"
  }

What did I miss to have the Service Name in APM ?

GregKalapos · August 8, 2022, 1:47pm

Hi @Nicolas_Rey,

the service name is not yet automatically added by the .NET APM Agent to the log line. This is something we already discuss to implement (not timeline at this point).

Currently the agent adds trace id and transaction id to log lines, but not the service name.

One thing you may be able to do is to manually add it (I know, this is less than ideal).

You can query the service name by looking at the currently active transaction:

Agent.Tracer?.CurrentTransaction?.Context.Service.Name

At this point it's probably better to just wait for an implementation of this.

Nicolas_Rey · August 21, 2022, 4:15pm

Thank you for you quick answer @GregKalapos
If I need to set it manually, I'm supposed to set the field you mentioned ?

Agent.Tracer?.CurrentTransaction?.Context.Service.Name

When I try to access to it, it says that Service is not part of Context, using the following APM version

<PackageReference Include="Elastic.Apm.NetCoreAll" Version="1.16.1" />

And could we set it globally, not only for the current span running ?
Do you have any idea of the roadmap for this implementation ?

Good vacations if you have some

GregKalapos · August 25, 2022, 5:39pm

Hi @Nicolas_Rey,

I just realize it's an internal property - sorry about that. My original answer wasn't really any help. Also that property is only set if we want to overwrite the default service name on a transaction level, in default cases it's empty and the agent sends it in the metadata and not on the transaction.

Anyways... second idea: You could use Agent.Config.ServiceName - that'll also return the service name. Plus, you can read it always, the other one would have required an active transaction - with this you can set it to all the log lines, also on ones where the agent does not have an active transaction.

And could we set it globally, not only for the current span running ?

I think this won't be an issue with the new idea.

Do you have any idea of the roadmap for this implementation ?

Nothing specific, but we are working on the ecs-dotnet repo and we really want to make progress on it. But no specific time frame.

Good vacations if you have some

Thank you!

Nicolas_Rey · August 29, 2022, 11:58am

Thank you @GregKalapos.
Well I tested the field Agent.Config.ServiceName is correctly fed, it is with my MS name.
Did I misunderstood something ?

Sorry to bother you where you need time to focus on ecs-dotnet, if there is a quick win workaround I'd like to apply it but currently I'm not sure of what I'm supposed to do.

Wolfgang_Ziegler · August 29, 2022, 12:16pm

Hi @Nicolas_Rey,
I'm stepping in for @GregKalapos while he is on vacation.

Did I understand correctly that you see the expected value in Agent.Config.ServiceName?
If yes, I believe that Greg's suggestion was to add this value manually.
Would that work for you (as a workaround)?

Nicolas_Rey · August 29, 2022, 12:53pm

Hi @Wolfgang_Ziegler , well yes Agent.Config.ServiceName is filled in with the expected value (my service name), but on APM the Service Name is empty (see my top question).

Greg answered me that it's currently not implemented, but that I can do it manually. So I don't understand the link with Agent.Config.ServiceName which is filled in as expected. And btw the ServiceName field has only a getter, I would not be able to set it on the fly if it was not already filled in.

Wolfgang_Ziegler · August 29, 2022, 1:25pm

Correct, Agent.Config.ServiceName is read-only - that's how it's meant to be.
The suggestion by Greg, as far as I understood, is to use this property and manually (since this is only a workaround) set up a SerilogEnricher (like here e.g.) to report this property as service name so it shows up in the UI where you expected it.

Nicolas_Rey · August 29, 2022, 1:32pm

That's perfectly make sense, sorry for the confusion, I focused on APM rather on the log enricher which is the 'root cause' of my concern.
One last question, any idea of the exact property name/key expected ?

Wolfgang_Ziegler · August 29, 2022, 1:49pm

No worries, glad we could clear this up
As for the attribute name, I would assume ElasticApmServiceName - but maybe @Martijn_Laarman can help out here as the ECS expert?

Nicolas_Rey · August 30, 2022, 7:24am

@Wolfgang_Ziegler anyway I'll test with ElasticApmServiceName and come back to you.

Many thanks for your time, as usual I'm impressed by the reactivity and availability of the Elastic team, especially on the .net driver

Nicolas_Rey · August 31, 2022, 2:02pm

Hello @Wolfgang_Ziegler @Martijn_Laarman , I tested to enrich the log with ElasticApmServiceName, it adds the information in the log itself, but still can't see this information in the log panel

The code

            logEvent.AddPropertyIfAbsent(propertyFactory.CreateProperty(
                "ElasticApmServiceName", "test"));

Produces

So the ElasticApmServiceName is transformed into metadata.elastic_apm_service_name .
But my logs panel is still empty.

From another stack application (Apollo Graphql), I checked how the logs are filled in, because the service names appears in the logs panel

so the the expected information seems to be service.name

When I try to use it in the serilog enricher

            logEvent.AddPropertyIfAbsent(propertyFactory.CreateProperty(
                "service.name", "test"));

The field is enriched

But the logs panel service name column stays empty.

Do you have insight of the key I'm supposed to fill in the enricher, I'm a bit lost ?
Thank you for your help

Wolfgang_Ziegler · August 31, 2022, 3:53pm

Hi @Nicolas_Rey ,

I looked into this again and applied the solution I found here: Outputting service.name instead of / alongside context.service.name

With that, you can create an alias for the field you already have metadata.service.name to service.name.

I tested this with my sample application and it looks promising:

(note: I aliased from fields.service.name to service.name)

Maybe @Martijn_Laarman has a better solution but this is the workaround I can suggest at the moment.

Cheers,
Wolfgang

Martijn_Laarman · August 31, 2022, 6:32pm

Sadly [FEATURE] Enrich APM integrations with APM Service name · Issue #134 · elastic/ecs-dotnet · GitHub still needs addressing so we can do this properly out of the box.

An easier way to do this manually would be:

var config = new EcsTextFormatterConfiguration()
	.MapCustom((ecsDoc, log) =>
	{
		ecsDoc.Service = new Service { Name = Agent.Config.ServiceName };
		return ecsDoc;
	});
var formatter = new EcsTextFormatter(config);

The MapCustom callback allows you to manually mutate the ecs document before its written to Elasticsearch.

Then pass that to ElasticsearchSinkOptions

CustomFormatter = formatter

Nicolas_Rey · September 1, 2022, 8:17am

You're awesome @Wolfgang_Ziegler and @Martijn_Laarman , it works like a charm! (I implemented EcsTextFormatter solution)

Many thanks guys

Topic		Replies	Views
Correlation logs are not reported in background service APM docker , dotnet	4	1367	November 12, 2020
How can you correlate logs with APM? APM dotnet	8	1815	June 25, 2021
Logs Do Not Appear In APM, Although Correlation Info Is Present Kibana	14	5891	January 5, 2022
Elastic apm : can't find service name mentioned in elastic apm agent on kibana ui or in elastic index APM java	2	1047	February 7, 2020
Need help getting .Net APM client running APM docker , dotnet , server , language-clients	2	848	August 29, 2022

APM .net agent see the Service Name in the logs for correlated transactions

Related topics