# Autodiscover drop default filebeat index

**URL:** <https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 16, 2020, 7:27am UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237 "2020-06-16T07:27:50Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shiny\_Hou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shiny_hou/32/70435_2.png) [@Shiny\_Hou](https://discuss.elastic.co/u/Shiny_Hou)\
**Post date:** [June 16, 2020, 7:27am UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237/1 "2020-06-16T07:27:50Z")

</div>

I don't want need `filbeat-version-%{+yyyy.MM.dd}` Index, my config don't have the config ， but the container log still input to `filbeat-version-%{+yyyy.MM.dd}`

```auto
---
    filebeat.autodiscover:
      providers:
      - type: kubernetes
        node: ${NODE_NAME}
        hints.enabled: true
        templates:
        - condition:
            equals:
              kubernetes.namespace: dev
          config:
          - type: container
            paths: 
            - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
            fields:
              logtype: app-console
          - type: log
            paths:
            - /var/log/k8s/${data.kubernetes.pod.name}/*.log
            multiline:
              pattern: '^\d{4}-\d{2}-\d{2}'
              negate: true
              match: after
            fields:
              logtype: app-file
        - condition:
            equals:
              kubernetes.namespace: istio-system
          config:
            - type: log
              paths:
              - /var/log/istio-proxy/access.log
              fields:
                logtype: ingressgateway-access
        - config:
          - type: container
            paths:
            - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
            fields:
              logtype: kubernetes-pods

    processors:
    - drop_event:
        when:
          equals:
            kubernetes.container.name: 'istio-proxy'
    - add_kubernetes_metadata:
        default_indexers.enabled: false
        default_matchers.enabled: false
        in_cluster: true

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      pipelines:
      - pipeline: "ingressgateway-access"
        when.equals:
          fields.logtype: ingressgateway-access
      indices:
      - index: "app-console-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: app-console
      - index: "ingressgateway-access-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: ingressgateway-access
      - index: "app-file-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: app-file
      - index: "kubernetes-pods-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: kuberentes-pods

```

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [June 16, 2020, 5:52pm UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237/2 "2020-06-16T17:52:04Z")

</div>

Hey @Shiny_Hou, welcome to discuss 🙂

Since 7.7.0, indexes are managed by [ILM](https://www.elastic.co/guide/en/beats/filebeat/7.7/ilm.html). If you want to disable this behaviour and use your own policies you can disable ilm with `setup.ilm.enabled: false`.

---

<div class="post-metadata">

**Author:** ![Shiny\_Hou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shiny_hou/32/70435_2.png) [@Shiny\_Hou](https://discuss.elastic.co/u/Shiny_Hou)\
**Post date:** [June 17, 2020, 1:18am UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237/3 "2020-06-17T01:18:30Z")

</div>

thanks 🥰 , @jsoriano i setted `setup.ilm.enabled: false.` but still exists the filebeat index

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [June 17, 2020, 10:14am UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237/4 "2020-06-17T10:14:30Z")

</div>

The filebeat index won't be removed. But after disabling ILM, are events being written to this index or to the custom ones?

---

<div class="post-metadata">

**Author:** ![Shiny\_Hou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shiny_hou/32/70435_2.png) [@Shiny\_Hou](https://discuss.elastic.co/u/Shiny_Hou)\
**Post date:** [June 17, 2020, 12:08pm UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237/5 "2020-06-17T12:08:19Z")

</div>

disable ilm, delete all index, delete fielbeat , then renew apply filebat like this :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/0/30cde86f49e143aa06e6e04d39ddf073d412d217.png)

---

<div class="post-metadata">

**Author:** ![Shiny\_Hou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shiny_hou/32/70435_2.png) [@Shiny\_Hou](https://discuss.elastic.co/u/Shiny_Hou)\
**Post date:** [June 17, 2020, 12:10pm UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237/6 "2020-06-17T12:10:01Z")

</div>

this my filebeat config:

```auto
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    setup.ilm.enabled: false
    ilm.enabled: false
    output.elasticsearch.ilm.enabled: false
    queue.mem:
      events: 4096
      flush.min_events: 512
      flush.timeout: 5s
    filebeat.autodiscover:
      providers:
      - type: kubernetes
        node: ${NODE_NAME}
        hints.enabled: true
        templates:
        - condition:
            equals:
              kubernetes.namespace: dev
          config:
          - type: container
            paths:
            # - /var/log/containers/*${data.kubernetes.container.id}.log
            - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
            fields:
              logtype: app-console
          - type: log
            paths:
            - /var/log/k8s/${data.kubernetes.pod.name}/*.log
            multiline:
              pattern: '^\d{4}-\d{2}-\d{2}'
              negate: true
              match: after
            fields:
              logtype: app-file
        - condition:
            equals:
              kubernetes.namespace: istio-system
          config:
            - type: log
              paths:
              - /var/log/istio-proxy/access.log
              fields:
                logtype: ingressgateway-access
        - config:
          - type: container
            paths:
            - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
            fields:
              logtype: kubernetes-pods

    processors:
    - drop_event:
        when:
          equals:
            kubernetes.container.name: 'istio-proxy'
    - add_kubernetes_metadata:
        default_indexers.enabled: false
        default_matchers.enabled: false
        in_cluster: true

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      pipelines:
      - pipeline: "ingressgateway-access"
        when.equals:
          fields.logtype: ingressgateway-access
      indices:
      - index: "app-console-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: app-console
      - index: "ingressgateway-access-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: ingressgateway-access
      - index: "app-file-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: app-file
      - index: "kubernetes-pods-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: kuberentes-pods

---

```

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [June 17, 2020, 5:46pm UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237/7 "2020-06-17T17:46:08Z")

</div>

I see you have indexes starting with `app-console-`, `ingressgateway-` and so on, do they contain the events you expect?

---

<div class="post-metadata">

**Author:** ![Shiny\_Hou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shiny_hou/32/70435_2.png) [@Shiny\_Hou](https://discuss.elastic.co/u/Shiny_Hou)\
**Post date:** [June 18, 2020, 1:37am UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237/8 "2020-06-18T01:37:01Z")

</div>

yes , they are look good, but `/var/lib/docker/containers` all log stdout to `filbeat-7.7.1-xxxx`

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [June 18, 2020, 5:46pm UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237/9 "2020-06-18T17:46:35Z")

</div>

Oh, I have just seen that you have hints-based autodiscover enabled. When hints-based autodiscover is enabled in filebeat it collects logs from all pods by default.  
I would avoid using templates and hints at the same time unless really needed, they generate each one their own set of configurations and can be difficult to get it configured as expected.

Could you try to disable hints (`hints.enabled: false`) to see if filebeat does what you expect?

---

<div class="post-metadata">

**Author:** ![Shiny\_Hou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shiny_hou/32/70435_2.png) [@Shiny\_Hou](https://discuss.elastic.co/u/Shiny_Hou)\
**Post date:** [June 19, 2020, 6:05am UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237/10 "2020-06-19T06:05:14Z")

</div>

thanks ! i use this config , it's look good

```auto
setup.ilm.enabled: false
     filebeat.autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
        hints.enabled: true
         hints.default_config.enabled: false

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 17, 2020, 8:05am UTC](https://discuss.elastic.co/t/autodiscover-drop-default-filebeat-index/237237/11 "2020-07-17T08:05:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.