Kubernetes metadata through Serilog.Sinks.Elasticsearch

Elastic Observability APM
1

I'm using APM on all my microservices and sending logs to ES through Serilog (.NET microservices). In the APM transaction view page there are two options: Container logs and Pod logs which filters by the container/pod uid.
I'm asked to make those options work, even though Serilog is not currently bringing Kubernetes metadata in the logs. I know this can be easily achieved with Filebeat, but I would need it to work with Serilog.

Any clues on how to make this work will be much appreciated. Cheers!

2

I'll answer myself, as I found the way to achieve this after a lot of research. Maybe this can be useful to someone.

First thing you want to do is add a custom field to Serilog in Program.cs

Log.Logger = new LoggerConfiguration()
   .ReadFrom.Configuration(Configuration)
   .Enrich.WithProperty("k8suid", Configuration.GetValue<string>("KUBERNETES_POD_UID") )
   .CreateLogger();

As you can see, we create a new field called k8suid which gets its value from an environment variable called KUBERNETES_UID. Let's now edit the Kubernetes deployment and add the environment variable (as documented here):

- name: KUBERNETES_POD_UID
    valueFrom:
        fieldRef:
            fieldPath: metadata.uid

This environment variable will give Serilog the value of the pod UID and also will overwrite the UID APM takes (I don't know why but it seems to use a different format, maybe hashes it).

In this point we will have a field called "fields.k8suid". It will be given the prefix "fields", but we need the field to be called "kubernetes.pod.uid". To achieve this, we will use a very simple pipeline. You can read more about them here. Let's go to Management > Dev Tools in Kibana's left menu, and execute the following:

PUT _ingest/pipeline/kubernetes_metadata_pipeline
{
  "description" : "Rewrite my kubernetes metadata field name",
  "processors" : [
    {
      "rename" : {
        "field": "fields.k8suid",
        "target_field": "kubernetes.pod.uid"
      }
    }
  ]
}

Now, we need to point our API's Serilog to this pipeline. In appsettings.json add the following line in WriteTo > ElasticSearch > Args:
"pipelineName": "kubernetes_metadata_pipeline"

From now on, every log sent to Elasticsearch will be processed by this pipeline and have it's field renamed. Now we can use the button "Container logs" from the APM trace view. You could add more parameters, like node name, pod name and namespace, as documented in the first link.

3

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.