Configuration to process certain logs with pipeline using Autodiscover

[zvazquez (Zareh Vazquez)]

Hi,

I have filebeats running on Kubernetes with the autodiscover option, I have Ambassador running on the K8 cluster among several other application and I am trying to process the logs for the Ambassador containers differently using a pipeline on Elasticsearch to extract the details of the access logs using the following autodicover configuration:

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          add_resource_metadata:
            namespace:
              enabled: true
          hints.enabled: true
          templates:
            - condition:
                and:
                  - contains.kubernetes.container.name: ambassador
                  - contains.message: ACCESS
              config:
                - type: container
                  pipeline: ambassador-access-log
                  paths:
                    - /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
            - config:
                - type: container
                  paths:
                    - /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log

Sadly, this is not working and seems that I am loosing all the logs from the ambassador containers. Is what I am trying to do possible? what could be wrong in the configuration?

Thanks,

Zareh

[stephenb (Stephen Brown)]

Hi @zvazquez

I think there is a couple issues here.

First I think this and condition will never be true.

@zvazquez

                and:
                  - contains.kubernetes.container.name: ambassador
                  - contains.message: ACCESS

If you look at the fields that are available for autodiscover here they are only specific fields.

- contains.message: ACCESS

The messages have not been parse when this condition applies therefor there is no message field present so I do not think that will ever be true.

If you want to separate out access and error logs you will probably need to do it another way, perhaps another label or something like

          - condition:
                and:
                  - contains.kubernetes.container.name: ambassador
              config:
                - type: container
                  pipeline: ambassador-access-log
                  paths:
                    - /var/lib/docker/containers/*/${data.kubernetes.container.id}-access-json.log
  

2nd I think that second - config still needs a condition see here