How to configure Filebeat to recognize ECS data

Elastic Stack Beats
,
1

Hello

I have an Kibana Elastic Filebeat docker stack and I want to monitor the logs of some NodeJS microservices (that run in docker too). I discovered yesterday ecs-morgan-format and it seems to work well with morgan.

I don't really know how to config my filebeat.yml to recognize the ECS format that are in the logs.

Actually I tried that :

  providers:
    # Disabling monitoring of containers that have a label "filebeat.disable" set to true
    - type: docker
      #hints.enabled: true
      templates:
        - condition.and:
            - not.contains:
                docker.container.labels.filebeat.disable: "true"
            #Disabling also frontend monitoring so we don't have the logs twice with the nginx module
            - not.contains:
                docker.container.name: frontend
            - not.contains:
                docker.container.name: traefik
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              exclude_lines: ['^[[:space:]]*$']
              fields_under_root: true
          # processors:
          #   - decode_json_fields:

And the logs I see in kibana are ECS in a 'message' field as it seems Filebeat isn't configured to analyze the ECS and send it to Elasticsearch.

Edit: Here is an example of what I see in Kibana/Elastic (the second entry is good, I enabled a second output)

1542×255 61.1 KB

Thanks for your help :slight_smile:

2

Hi @Noxis, welcome to the community!

Reviewing and testing with the filebeat configuration snippet you provided, I saw the same results. The docker input configuration will need to be updated for your node containers to support the JSON decoding of the log lines:

json.keys_under_root: true
json.overwrite_keys: true
3

Thanks a lot for your answer. It's working now ! :slight_smile:

Here is my final config in case someone needs some help in the future :

filebeat.autodiscover:
  providers:
    # Disabling monitoring of containers that have a label "filebeat.disable" set to true
    - type: docker
      #hints.enabled: true
      templates:
        - condition.and:
            - not.contains:
                docker.container.labels.filebeat.disable: "true"
            #Disabling also frontend monitoring so we don't have the logs twice with the nginx module
            - not.contains:
                docker.container.name: frontend
            - not.contains:
                docker.container.name: traefik
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              exclude_lines: ['^[[:space:]]*$']
              fields_under_root: true
              json:
                keys_under_root: true
                message_key: message
                overwrite_keys: true
4

Just wanted to throw another round of thanks!

I think it should be clearer on the docs what annotations we need to put for services using ECS format.

Cheers