Autodiscover drop default filebeat index

Elastic Stack Beats
1

I don't want need filbeat-version-%{+yyyy.MM.dd} Index, my config don't have the config ， but the container log still input to filbeat-version-%{+yyyy.MM.dd}

---
    filebeat.autodiscover:
      providers:
      - type: kubernetes
        node: ${NODE_NAME}
        hints.enabled: true
        templates:
        - condition:
            equals:
              kubernetes.namespace: dev
          config:
          - type: container
            paths: 
            - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
            fields:
              logtype: app-console
          - type: log
            paths:
            - /var/log/k8s/${data.kubernetes.pod.name}/*.log
            multiline:
              pattern: '^\d{4}-\d{2}-\d{2}'
              negate: true
              match: after
            fields:
              logtype: app-file
        - condition:
            equals:
              kubernetes.namespace: istio-system
          config:
            - type: log
              paths:
              - /var/log/istio-proxy/access.log
              fields:
                logtype: ingressgateway-access
        - config:
          - type: container
            paths:
            - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
            fields:
              logtype: kubernetes-pods

    processors:
    - drop_event:
        when:
          equals:
            kubernetes.container.name: 'istio-proxy'
    - add_kubernetes_metadata:
        default_indexers.enabled: false
        default_matchers.enabled: false
        in_cluster: true

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      pipelines:
      - pipeline: "ingressgateway-access"
        when.equals:
          fields.logtype: ingressgateway-access
      indices:
      - index: "app-console-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: app-console
      - index: "ingressgateway-access-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: ingressgateway-access
      - index: "app-file-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: app-file
      - index: "kubernetes-pods-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: kuberentes-pods
2

Hey @Shiny_Hou, welcome to discuss :slight_smile:

Since 7.7.0, indexes are managed by ILM. If you want to disable this behaviour and use your own policies you can disable ilm with setup.ilm.enabled: false.

3

thanks :smiling_face_with_three_hearts: , @jsoriano i setted setup.ilm.enabled: false. but still exists the filebeat index

4

The filebeat index won't be removed. But after disabling ILM, are events being written to this index or to the custom ones?

5

disable ilm, delete all index, delete fielbeat , then renew apply filebat like this :

image
image587×501 36.1 KB

6

this my filebeat config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    setup.ilm.enabled: false
    ilm.enabled: false
    output.elasticsearch.ilm.enabled: false
    queue.mem:
      events: 4096
      flush.min_events: 512
      flush.timeout: 5s
    filebeat.autodiscover:
      providers:
      - type: kubernetes
        node: ${NODE_NAME}
        hints.enabled: true
        templates:
        - condition:
            equals:
              kubernetes.namespace: dev
          config:
          - type: container
            paths:
            # - /var/log/containers/*${data.kubernetes.container.id}.log
            - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
            fields:
              logtype: app-console
          - type: log
            paths:
            - /var/log/k8s/${data.kubernetes.pod.name}/*.log
            multiline:
              pattern: '^\d{4}-\d{2}-\d{2}'
              negate: true
              match: after
            fields:
              logtype: app-file
        - condition:
            equals:
              kubernetes.namespace: istio-system
          config:
            - type: log
              paths:
              - /var/log/istio-proxy/access.log
              fields:
                logtype: ingressgateway-access
        - config:
          - type: container
            paths:
            - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
            fields:
              logtype: kubernetes-pods

    processors:
    - drop_event:
        when:
          equals:
            kubernetes.container.name: 'istio-proxy'
    - add_kubernetes_metadata:
        default_indexers.enabled: false
        default_matchers.enabled: false
        in_cluster: true

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      pipelines:
      - pipeline: "ingressgateway-access"
        when.equals:
          fields.logtype: ingressgateway-access
      indices:
      - index: "app-console-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: app-console
      - index: "ingressgateway-access-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: ingressgateway-access
      - index: "app-file-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: app-file
      - index: "kubernetes-pods-%{+yyyy.MM.dd}"
        when.equals:
          fields.logtype: kuberentes-pods

---
7

I see you have indexes starting with app-console-, ingressgateway- and so on, do they contain the events you expect?

8

yes , they are look good, but /var/lib/docker/containers all log stdout to filbeat-7.7.1-xxxx

9

Oh, I have just seen that you have hints-based autodiscover enabled. When hints-based autodiscover is enabled in filebeat it collects logs from all pods by default.
I would avoid using templates and hints at the same time unless really needed, they generate each one their own set of configurations and can be difficult to get it configured as expected.

Could you try to disable hints (hints.enabled: false) to see if filebeat does what you expect?

1 Like
10

thanks ! i use this config , it's look good

setup.ilm.enabled: false
     filebeat.autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
        hints.enabled: true
         hints.default_config.enabled: false
1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.