# Configuration to process certain logs with pipeline using Autodiscover

**URL:** <https://discuss.elastic.co/t/configuration-to-process-certain-logs-with-pipeline-using-autodiscover/274407>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 29, 2021, 5:35pm UTC](https://discuss.elastic.co/t/configuration-to-process-certain-logs-with-pipeline-using-autodiscover/274407 "2021-05-29T17:35:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![zvazquez](https://avatars.discourse-cdn.com/v4/letter/z/c0e974/32.png) [@zvazquez](https://discuss.elastic.co/u/zvazquez)\
**Post date:** [May 29, 2021, 5:35pm UTC](https://discuss.elastic.co/t/configuration-to-process-certain-logs-with-pipeline-using-autodiscover/274407/1 "2021-05-29T17:35:01Z")

</div>

Hi,

I have filebeats running on Kubernetes with the autodiscover option, I have Ambassador running on the K8 cluster among several other application and I am trying to process the logs for the Ambassador containers differently using a pipeline on Elasticsearch to extract the details of the access logs using the following autodicover configuration:

```auto
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          add_resource_metadata:
            namespace:
              enabled: true
          hints.enabled: true
          templates:
            - condition:
                and:
                  - contains.kubernetes.container.name: ambassador
                  - contains.message: ACCESS
              config:
                - type: container
                  pipeline: ambassador-access-log
                  paths:
                    - /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
            - config:
                - type: container
                  paths:
                    - /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log

```

Sadly, this is not working and seems that I am loosing all the logs from the ambassador containers. Is what I am trying to do possible? what could be wrong in the configuration?

Thanks,

Zareh

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 29, 2021, 6:51pm UTC](https://discuss.elastic.co/t/configuration-to-process-certain-logs-with-pipeline-using-autodiscover/274407/2 "2021-05-29T18:51:49Z")

</div>

Hi @zvazquez

I think there is a couple issues here.

First I think this and condition will never be true.

> [@zvazquez](#):
>
> ```auto
> and:
> - contains.kubernetes.container.name: ambassador
> - contains.message: ACCESS
> 
> ```

If you look at the fields that are available for autodiscover [here](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html) they are only specific fields.

`- contains.message: ACCESS`

The messages have not been parse when this condition applies therefor there is no message field present so I do not think that will ever be true.

If you want to separate out access and error logs you will probably need to do it another way, perhaps another label or something like

```auto
          - condition:
                and:
                  - contains.kubernetes.container.name: ambassador
              config:
                - type: container
                  pipeline: ambassador-access-log
                  paths:
                    - /var/lib/docker/containers/*/${data.kubernetes.container.id}-access-json.log
  

```

2nd I think that second` - config` still needs a `condition` see [here](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html#_parsing_json_logs)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 26, 2021, 8:51pm UTC](https://discuss.elastic.co/t/configuration-to-process-certain-logs-with-pipeline-using-autodiscover/274407/3 "2021-06-26T20:51:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.