Filebeat, ECS and fieds.* namespace

[A_B]

Hello all,

I'm looking for some suggestions for the following situation.

Some context
I'm trying to convince our developers to adopt ECS as their log format for container logs.

We run our containers in Kubernetes and use Filebeat as a DaemonSet to collect all container logs.

Filebeat config

- type: container
  fields:
    log_prefix: k8s
    log_idx: ${K8S_CLUSTER}
  fields_under_root: false
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
  multiline.negate: false
  multiline.match: after
  paths:
    - "/var/lib/docker/containers/*/*.log"
  processors:
    - add_kubernetes_metadata:
        in_cluster: true
    - decode_json_fields:
        add_error_key: true
        overwrite_keys: true
        fields: ["message"]
        target: ""
        when:
          equals:
            kubernetes.labels.logFormat: "ecs"

We are adding a couple of fields later used by Logstash to route the documents to the correct Elasticsearch indices.

For the decode_json_fields processor we have set overwrite_keys so that in the final document e.g. message is overwritten by the ECS message field from the container log.

This has worked quite well until the first logs containing fields.* came in and because of overwrite_keys, those fields over write the fields added by Filebeat.

I know I can put the additional fields Filebeat is adding in the root of the JSON. Is that the best option for me? I can't see fields.* as part of ECS. Does anyone know if it is?

Any other options for me that anyone can think of?

Cheers,
AB

[ebeahan (Eric Beahan)]

You're correct that fields.* is not a defined fieldset in ECS.

Another possibility: use the add_fields processor and place your fields into a namespace that aligns with the ECS best practices for custom fields.

processors:
  - add_fields:
      target: My_Custom_Namespace
      fields:
        log_prefix: k8s
        log_idx: ${K8S_CLUSTER}

[A_B]

Hi @ebeahan,

thank you very much for your reply and your suggestion.