# How to exclude other namespaces?

**URL:** <https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [August 27, 2020, 4:00am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544 "2020-08-27T04:00:02Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [August 27, 2020, 4:00am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/1 "2020-08-27T04:00:02Z")

</div>

> ```
> filebeat.autodiscover:
> providers:
> - type: kubernetes
> node: ${NODE_NAME}
> hints.enabled: false
> hints.default_config:
> type: container
> finished: true
> paths:
> - "/var/log/containers/*.log"
> templates:
> - condition:
> or:
> - equals:
> kubernetes.namespace: "front"
> - equals:
> kubernetes.namespace: "back"
> config:
> - type: container
> paths:
> - "/var/log/containers/*.log"
> setup.template.enabled: false
> processors:
> - add_cloud_metadata:
> - add_host_metadata:
> - add_kubernetes_metadata:
> in_cluster: true
> host: ${NODE_NAME}
> default_matchers.enabled: false
> matchers:
> - logs_path:
> logs_path: "/var/log/containers/"
> resource_type: "container"
> 
> output.elasticsearch:
> hosts: ${ELASTICSEARCH_HOST}
> 
> ```

I only need the logs under the back and front namespaces.  
The above method did not succeed. How should I modify it?

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [August 27, 2020, 6:51am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/2 "2020-08-27T06:51:51Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/8/a/8a81144e3d9392f820e35ac1cbcc34a4aa5a8df4.png)  
In addition, I have a question, why filebeat will inject information into application events.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 27, 2020, 11:23am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/3 "2020-08-27T11:23:02Z")

</div>

Hey @wajika,

In your configuration you are using a path with a wildcard that would match all the containers in the node. So every configuration generated, for every pod, will try to harvest any file. You need to setup autodiscover in a way that it generates an specific configuration for each container. Also, when using autodiscover, you don't need to use `add_kubernetes_metadata`, events should be already enriched by the autodiscover provider.

I think that something like this would work for you:

```auto
filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            or:
              - equals:
                  kubernetes.namespace: "front"
              - equals:
                  kubernetes.namespace: "back"
          config:
            - type: container
              paths:
                - "/var/log/containers/*-${data.kubernetes.container.id}.log"
setup.template.enabled: false
processors:
  - add_cloud_metadata:
  - add_host_metadata:

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOST}

```

Other option could be to use hints-based autodiscover, that would allow you to enable collection of logs per namespace using annotations in the namespaces themselves. See my comment about that on this topic: [Collect logs for specific containers or namespace in Openshift/Kubernetes](https://discuss.elastic.co/t/collect-logs-for-specific-containers-or-namespace-in-openshift-kubernetes/232417/2)

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [August 28, 2020, 12:31am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/4 "2020-08-28T00:31:12Z")

</div>

> ```
> filebeat.autodiscover:
> providers:
> - type: kubernetes
> node: ${NODE_NAME}
> templates:
> - condition:
> or:
> - equals:
> kubernetes.namespace: "front"
> - equals:
> kubernetes.namespace: "back"
> config:
> - type: container
> paths:
> - "/var/log/containers/*-${data.kubernetes.container.id}.log"
> multiline.pattern: '^[[:space:]]'
> multiline.negate: false
> multiline.match: after
> 
> setup.template.enabled: false
> processors:
> - add_cloud_metadata:
> - add_host_metadata:
> 
> output.elasticsearch:
> hosts: ${ELASTICSEARCH_HOST}
> 
> ```

I tried to use your configuration, but filebeat reported an error.

> ```
> ERROR [autodiscover] autodiscover/autodiscover.go:209 Auto discover config check failed for config '{
> "docker-json": {
> "cri_flags": true,
> "format": "auto",
> "partial": true,
> "stream": "all"
> },
> "multiline": {
> "match": "after",
> "negate": false,
> "pattern": "^[[:space:]]"
> },
> "symlinks": true,
> "type": "container"
> }', won't start runner: each input must have at least one path defined
> 
> ```

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/e/6e0e8cfbbf4eac92eac4be65b0f95ce05ea186d7.png)

One more thing, if I use ${data.kubernetes.container.id}, can the container.id field be generated?  
I need to use a field to contact other beats data (APM and metricbeat) on the kubernetes cluster

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [August 28, 2020, 1:08am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/5 "2020-08-28T01:08:38Z")

</div>

> filebeat.autodiscover:  
> providers:  
> - type: kubernetes  
> hints.enabled: true  
> add\_resource\_metadata:  
> namespace:  
> enabled: true

If I use this configuration, can filebeat support both pod annotations and node annotations?

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [August 28, 2020, 5:26am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/6 "2020-08-28T05:26:49Z")

</div>

I feel that using the "co.elastic.logs/enabled:'false'" method to turn off the log collection of the namespace is not optimal. If there are a lot of namespaces, then I must increase them one by one.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 30, 2020, 11:28am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/7 "2020-08-30T11:28:21Z")

</div>

I think the error can be ignored in your case, but you can add a condition to ignore events without container ids, so the error doesn't happen.

Try this configuration:

```auto
filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            and:
              - has_fields: ['kubernetes.container.id']
              - or:
                - equals:
                    kubernetes.namespace: "front"
                - equals:
                    kubernetes.namespace: "back"
          config:
            - type: container
              paths:
                - "/var/log/containers/*-${data.kubernetes.container.id}.log"
setup.template.enabled: false
processors:
  - add_cloud_metadata:
  - add_host_metadata:

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOST}

```

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 30, 2020, 11:33am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/8 "2020-08-30T11:33:56Z")

</div>

> [@wajika](#):
>
> If I use this configuration, can filebeat support both pod annotations and node annotations?

Try with this to add node annotations:

```auto
      add_resource_metadata:
        node:
          enabled: true
          include_annotations:
            - "someannotation"
            - "someotherannotation"

```

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 30, 2020, 11:35am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/9 "2020-08-30T11:35:07Z")

</div>

> [@wajika](#):
>
> I feel that using the "co.elastic.logs/enabled:'false'" method to turn off the log collection of the namespace is not optimal. If there are a lot of namespaces, then I must increase them one by one.

Take into account that you wouldn't need to change the configuration, you would only need to add `co.elastic.logs/enabled:'true'` annotation to the namespaces you want to collect logs from.

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [August 31, 2020, 1:07am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/10 "2020-08-31T01:07:32Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/a/6/a6fc6ddbb09d2a5189adf9f73ca617819acd6ee4.png)  
I added has\_ fields: [' kubernetes.container.id '] still reporting errors.  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/e/8e80aee0fe0f9dd2e2fefc1a8124a1cbc267b265.png)

However, if you say that this error is not a big problem, let it be for the time being.

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [August 31, 2020, 1:09am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/11 "2020-08-31T01:09:20Z")

</div>

OK, that's easy.

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [August 31, 2020, 1:33am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/12 "2020-08-31T01:33:34Z")

</div>

Sorry, I didn't understand you.  
I also need a field container.id, can you advice me, how to do that?

Thank you for your reply.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 31, 2020, 9:00am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/13 "2020-08-31T09:00:30Z")

</div>

> [@wajika](#):
>
> I also need a field container.id, can you advice me, how to do that?

What do you mean? Filebeat should be adding the `kubernetes.container.id` field to collected logs.

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [August 31, 2020, 9:26am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/14 "2020-08-31T09:26:11Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/4/a/4ac81c8bde7fa4ffe26d6910f82125a2d60bd13f.png)  
Kubernetes matedata all appeared, but not yet container.id.  
kubernetes.container.id Where should field be added?

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 31, 2020, 9:38am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/15 "2020-08-31T09:38:38Z")

</div>

Umm, is this container being stopped at this moment? Looking at the code the only case where it seems possible to have the container name but not its id is when the pod is being stopped: [https://github.com/elastic/beats/blob/7fbbdca91b5cdfcb943ff7f7b7312219ae9986c0/libbeat/autodiscover/providers/kubernetes/pod.go#L339](https://github.com/elastic/beats/blob/7fbbdca91b5cdfcb943ff7f7b7312219ae9986c0/libbeat/autodiscover/providers/kubernetes/pod.go#L339)

If this is a normal running container this may be a bug. Are you missing the `kubernetes.container.id` in all events?

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [September 1, 2020, 1:02am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/16 "2020-09-01T01:02:39Z")

</div>

I can confirm that all containers have no container.id field.

Not even in the mapping.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/7/27247f8d854f34506c583c70fe316e784f63e481.png)

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [September 3, 2020, 1:04am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/17 "2020-09-03T01:04:08Z")

</div>

Is there a solution to this problem?

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [September 3, 2020, 9:23am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/18 "2020-09-03T09:23:28Z")

</div>

This seems unexpected to me, could you try with a released version, like 7.9.0?

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [September 4, 2020, 12:52am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/19 "2020-09-04T00:52:30Z")

</div>

This missing container field also occurs on APM agent (without kubernetes metadata).

Filebeat 7.9.1 will also not be generated container.id.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [September 4, 2020, 10:48am UTC](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544/20 "2020-09-04T10:48:15Z")

</div>

Hey @wajika,

I have tried to reproduce this and there seems to be actually some problem on Beats with the container ids. I have opened an issue in Github for further investigation: [https://github.com/elastic/beats/issues/20982](https://github.com/elastic/beats/issues/20982)

Thanks!

[Next page](https://discuss.elastic.co/t/how-to-exclude-other-namespaces/246544.md?page=2)