Guidance on ASP.NET core, Serilog and how to correctly log to an Elastic stack

Elastic Observability
1

We have a microservice architecture running in a kubernetes cluster, and are using the Elastic stack (deployed using EKS providers) for monitoring everything. One of the last aspects I'm struggling with, is how to correctly log to to elastic search, from an ASP.NET Core 5.x microservice.

If I understand correctly, I can log directly to Elastic Search using a combination of Serilog, Elastic search sink for serilog, and the elastic common schema (ESC) package.

I've setup Serilog according to the docs, like so:

Log.Logger = new LoggerConfiguration()
.Enrich.WithElasticApmCorrelationInfo()
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
{
  CustomFormatter = new EcsTextFormatter()
})
.CreateLogger();

This results in logs being sent to a logstash-{yyyy}-{mm}-{dd} index.
This however will not show those logs in the default 'Observability > Logs' view in Kibana, since the index pattern for that screen is set to logs-*,filebeat-*,kibana_sample_data_logs* by default.

So I thought I would need to send the logs to a logs-* index, but found out those are 'data streams' instead of plain old indices. Reading up on data streams, I thought it would be better to send logs to a data stream vs a plain old index, because it should offer better performance (automatic index splitting, vs having 1 index per service, per calendar day), so I modified my serilog elastic sink configuration as follows:

Log.Logger = new LoggerConfiguration()
.Enrich.WithElasticApmCorrelationInfo()
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
{
    AutoRegisterTemplate = true,
    IndexFormat = $"logs-my-micro-service",
    BatchAction = ElasticOpType.Create,
    CustomFormatter = new EcsTextFormatter()
})
.CreateLogger();

This seems to work fine at first glance, logs are correctly arriving in the data stream index, and are visible with the default 'Logs' screen configuration.
However, with unhandled exception logs, the logs sent to the data stream index are missing the error.stack_trace (string type) property.

I'm at a loss as to where to find the source of this problem, I'm also not sure whether the configuration I want to use (using data streams) is the recommended option.

2

Did you ever find a resolution to this? I'm looking at using data streams instead of directly to indexes and having a similar issue.

3

No, we resorted to sticking with plain old indices for the time being...

4

So to follow up on this, we got it working

First step: Create an index template as a datastream and whatever settings you want. Be sure the pattern will catch the indexes. If you using my-app-logs as the datastream I would set the pattern to be both my-app-logs and my-app-logs-*. The non-wildcard pattern may not be necessary, but this worked and I didn't want to redo everything today to verify. For us, we just let serilog create it's first template to copy over the settings to the datastream index template we are going to be using.

Second step: I like to use api keys so generate an api key as follows in Dev Tools within Kibana (if using curl, just modify as you need). The key here are the minimum index permissions for it work.

POST /_security/api_key
{
  "name": "logs-application",
  "role_descriptors": {
    "role-my-app-logs": {
      "index": [
        {
          "names": ["my-app-logs","my-app-logs-*"],
          "privileges": ["auto_configure","create_index","create_doc","read"]
        }
      ]
    }

  }
}

Be sure to gather the parts of the key you need. In the 3rd step here we use the id and api_key directly in the elasticsearchsinkoptions. If you're using the connectionHeader in appsettings or something else you need to use the encoded api key

Third Step: As this is using ModifyConnectionSettings instead of the header, your code may vary if you use headers

.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(esUri))
{
  //AutoRegisterTemplate is omitted as it defaults to false. Either remove from your config or change the 'true' value to 'false' as the template it creates is not a datastream and you use IndexFormat below to point to it
  //TypeName needs to be null
  TypeName = null,
  //IndexFormat needs to be the name of your datastream set in the index template
  IndexFormat = "my-app-logs",
  BatchAction = ElasticOpType.Create,
  ModifyConnectionSettings = x =>
  {
    return x.ApiKeyAuthentication("API_KEY_ID", "API_KEY");
  }
})

I hope this helps and sorry for the delay but we finally got around to focusing on getting this working.