How to configure Filebeat to recognize ECS data

Hello

I have an Kibana Elastic Filebeat docker stack and I want to monitor the logs of some NodeJS microservices (that run in docker too). I discovered yesterday ecs-morgan-format and it seems to work well with morgan.

I don't really know how to config my filebeat.yml to recognize the ECS format that are in the logs.

Actually I tried that :

  providers:
    # Disabling monitoring of containers that have a label "filebeat.disable" set to true
    - type: docker
      #hints.enabled: true
      templates:
        - condition.and:
            - not.contains:
                docker.container.labels.filebeat.disable: "true"
            #Disabling also frontend monitoring so we don't have the logs twice with the nginx module
            - not.contains:
                docker.container.name: frontend
            - not.contains:
                docker.container.name: traefik
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              exclude_lines: ['^[[:space:]]*$']
              fields_under_root: true
          # processors:
          #   - decode_json_fields:

And the logs I see in kibana are ECS in a 'message' field as it seems Filebeat isn't configured to analyze the ECS and send it to Elasticsearch.

Edit: Here is an example of what I see in Kibana/Elastic (the second entry is good, I enabled a second output)

Thanks for your help :slight_smile:

Hi @Noxis, welcome to the community!

Reviewing and testing with the filebeat configuration snippet you provided, I saw the same results. The docker input configuration will need to be updated for your node containers to support the JSON decoding of the log lines:

json.keys_under_root: true
json.overwrite_keys: true
3 Likes

Thanks a lot for your answer. It's working now ! :slight_smile:

Here is my final config in case someone needs some help in the future :

filebeat.autodiscover:
  providers:
    # Disabling monitoring of containers that have a label "filebeat.disable" set to true
    - type: docker
      #hints.enabled: true
      templates:
        - condition.and:
            - not.contains:
                docker.container.labels.filebeat.disable: "true"
            #Disabling also frontend monitoring so we don't have the logs twice with the nginx module
            - not.contains:
                docker.container.name: frontend
            - not.contains:
                docker.container.name: traefik
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              exclude_lines: ['^[[:space:]]*$']
              fields_under_root: true
              json:
                keys_under_root: true
                message_key: message
                overwrite_keys: true

Just wanted to throw another round of thanks!

I think it should be clearer on the docs what annotations we need to put for services using ECS format.

Cheers

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.