APM .net agent see the Service Name in the logs for correlated transactions

Kibana version:
7.16.3 (Docker)

Elasticsearch version:
7.16.3 (Docker)

APM Server version:
7.16.3 (Docker)

APM Agent language and version:
.net

    <PackageReference Include="Elastic.Apm.NetCoreAll" Version="1.16.1" />
    <PackageReference Include="Elastic.Apm.SerilogEnricher" Version="1.5.3" />
    <PackageReference Include="Elastic.Apm.StackExchange.Redis" Version="1.16.1" />

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Hi,
I use correlated transactions across multiple microservices (.net 6) using the .net APM agent, it works pretty well.
My problem is about logs, I don't know why the Service Name does not appear, so that I'm not able to distinguish which log belongs to which microservice.

Example here is an example of a correlated transaction

and the corresponding logs, but with an empty Service Name column

My Serilog logger instance is built this way :

        public static Logger GetELKLogger(IConfiguration config, string varEnv = "ElasticSearchLog")
        {
            var configuration = config.Get<Configuration>(varEnv);

            return new LoggerConfiguration()
            .ReadFrom.Configuration(config)
            .Enrich.WithElasticApmCorrelationInfo()
            .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(configuration.ElasticSearchLog))
            {
                AutoRegisterTemplate = true,
                IndexFormat = "mslogs-{0:yyyy.MM.dd}",
                DetectElasticsearchVersion = true,
                RegisterTemplateFailure = RegisterTemplateRecovery.IndexAnyway,
                AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
                FailureCallback = e => Console.WriteLine($"Unable to submit event {e?.RenderMessage()} to ElasticSearch. Exception : " + e?.Exception?.ToString()),
                EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
                                        EmitEventFailureHandling.WriteToFailureSink |
                                        EmitEventFailureHandling.RaiseCallback,
                BufferCleanPayload = (failingEvent, statuscode, exception) =>
                {
                    dynamic e = JObject.Parse(failingEvent);
                    return JsonConvert.SerializeObject(new Dictionary<string, object>()
                        {
                            { "action", "DeniedByElasticSearch"},
                            { "@timestamp",e["@timestamp"]},
                            { "level","Error"},
                            { "message","Error: "+e.message},
                            { "messageTemplate",e.messageTemplate},
                            { "failingStatusCode", statuscode},
                            { "failingException", exception}
                        });
                },
                CustomFormatter = new EcsTextFormatter()
            })
            .CreateLogger();
        }

My minimal appsettings.json for each Microservice is

  "Serilog": {
    "Using": [],
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Information",
        "Elastic": "Warning",
        "Apm": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Console"
      }
    ],
    "Enrich": [
      "FromLogContext",
      "WithMachineName",
      "WithProcessId",
      "WithThreadId"
    ],
    "Properties": {
      "ApplicationName": "IT.Microservices.AbandonedCartEmailSender"
    }
  },
  "ElasticApm": {
    "ServerUrl":"http://apm:8200",
    "Enabled": true,
    "TransactionSampleRate": 1,
    "CaptureBody": "all",
    "CaptureHeaders": true,
    "SpanFramesMinDuration": 0, // no stacktrace except for exception
    "CloudProvider": "none"
  },
  "ElasticSearchLog": {
    "ElasticSearchLog": "http://elasticsearch:9200/"
  }

What did I miss to have the Service Name in APM ?

Hi @Nicolas_Rey,

the service name is not yet automatically added by the .NET APM Agent to the log line. This is something we already discuss to implement (not timeline at this point).

Currently the agent adds trace id and transaction id to log lines, but not the service name.

One thing you may be able to do is to manually add it (I know, this is less than ideal).

You can query the service name by looking at the currently active transaction:

Agent.Tracer?.CurrentTransaction?.Context.Service.Name

At this point it's probably better to just wait for an implementation of this.

Thank you for you quick answer @GregKalapos
If I need to set it manually, I'm supposed to set the field you mentioned ?

Agent.Tracer?.CurrentTransaction?.Context.Service.Name

When I try to access to it, it says that Service is not part of Context, using the following APM version

<PackageReference Include="Elastic.Apm.NetCoreAll" Version="1.16.1" />

And could we set it globally, not only for the current span running ?
Do you have any idea of the roadmap for this implementation ?

Good vacations if you have some :palm_tree:

Hi @Nicolas_Rey,

I just realize it's an internal property - sorry about that. My original answer wasn't really any help. Also that property is only set if we want to overwrite the default service name on a transaction level, in default cases it's empty and the agent sends it in the metadata and not on the transaction.

Anyways... second idea: You could use Agent.Config.ServiceName - that'll also return the service name. Plus, you can read it always, the other one would have required an active transaction - with this you can set it to all the log lines, also on ones where the agent does not have an active transaction.

And could we set it globally, not only for the current span running ?

I think this won't be an issue with the new idea.

Do you have any idea of the roadmap for this implementation ?

Nothing specific, but we are working on the ecs-dotnet repo and we really want to make progress on it. But no specific time frame.

Good vacations if you have some :palm_tree:

Thank you! :slight_smile:

Thank you @GregKalapos.
Well I tested the field Agent.Config.ServiceName is correctly fed, it is with my MS name.
Did I misunderstood something ?

Sorry to bother you where you need time to focus on ecs-dotnet, if there is a quick win workaround I'd like to apply it but currently I'm not sure of what I'm supposed to do.

Hi @Nicolas_Rey,
I'm stepping in for @GregKalapos while he is on vacation.

Did I understand correctly that you see the expected value in Agent.Config.ServiceName?
If yes, I believe that Greg's suggestion was to add this value manually.
Would that work for you (as a workaround)?

Hi @Wolfgang_Ziegler , well yes Agent.Config.ServiceName is filled in with the expected value (my service name), but on APM the Service Name is empty (see my top question).

Greg answered me that it's currently not implemented, but that I can do it manually. So I don't understand the link with Agent.Config.ServiceName which is filled in as expected. And btw the ServiceName field has only a getter, I would not be able to set it on the fly if it was not already filled in.

Correct, Agent.Config.ServiceName is read-only - that's how it's meant to be.
The suggestion by Greg, as far as I understood, is to use this property and manually (since this is only a workaround) set up a SerilogEnricher (like here e.g.) to report this property as service name so it shows up in the UI where you expected it.

That's perfectly make sense, sorry for the confusion, I focused on APM rather on the log enricher which is the 'root cause' of my concern.
One last question, any idea of the exact property name/key expected ?

No worries, glad we could clear this up :slight_smile:
As for the attribute name, I would assume ElasticApmServiceName - but maybe @Martijn_Laarman can help out here as the ECS expert?

1 Like

@Wolfgang_Ziegler anyway I'll test with ElasticApmServiceName and come back to you.

Many thanks for your time, as usual I'm impressed by the reactivity and availability of the Elastic team, especially on the .net driver :raised_hands:

Hello @Wolfgang_Ziegler @Martijn_Laarman , I tested to enrich the log with ElasticApmServiceName, it adds the information in the log itself, but still can't see this information in the log panel

The code

            logEvent.AddPropertyIfAbsent(propertyFactory.CreateProperty(
                "ElasticApmServiceName", "test"));

Produces

So the ElasticApmServiceName is transformed into metadata.elastic_apm_service_name .
But my logs panel is still empty.

From another stack application (Apollo Graphql), I checked how the logs are filled in, because the service names appears in the logs panel

so the the expected information seems to be service.name

When I try to use it in the serilog enricher

            logEvent.AddPropertyIfAbsent(propertyFactory.CreateProperty(
                "service.name", "test"));

The field is enriched

But the logs panel service name column stays empty.

Do you have insight of the key I'm supposed to fill in the enricher, I'm a bit lost ?
Thank you for your help

Hi @Nicolas_Rey ,

I looked into this again and applied the solution I found here: Outputting service.name instead of / alongside context.service.name

With that, you can create an alias for the field you already have metadata.service.name to service.name.

I tested this with my sample application and it looks promising:

(note: I aliased from fields.service.name to service.name)

Maybe @Martijn_Laarman has a better solution but this is the workaround I can suggest at the moment.

Cheers,
Wolfgang

Sadly [FEATURE] Enrich APM integrations with APM Service name · Issue #134 · elastic/ecs-dotnet · GitHub still needs addressing so we can do this properly out of the box.

An easier way to do this manually would be:

var config = new EcsTextFormatterConfiguration()
	.MapCustom((ecsDoc, log) =>
	{
		ecsDoc.Service = new Service { Name = Agent.Config.ServiceName };
		return ecsDoc;
	});
var formatter = new EcsTextFormatter(config);

The MapCustom callback allows you to manually mutate the ecs document before its written to Elasticsearch.

Then pass that to ElasticsearchSinkOptions

CustomFormatter = formatter
2 Likes

You're awesome @Wolfgang_Ziegler and @Martijn_Laarman , it works like a charm! (I implemented EcsTextFormatter solution)

Many thanks guys

1 Like