Creating indexes and ILM based on Kubernetes namespace using hints-based autodiscover

Hi all,

We're just getting started with Filebeat on Kubernetes and the Elastic Stack in general. I'm really impressed with the capabilities of Filebeat so far. The hints-based autodiscover feature is a big deal for us as it allows individual app teams to set their own match patterns and other Filebeat configuration settings based on annotations they place on their pods.

What I can't figure out, though, is how we can tell Filebeat (and by extension Elasticsearch) to create separate indexes based on Kubernetes namespaces when using hints/annotations. At the same time, we also want ILM to function for each index. I've played around with conditions and setup.ilm.x settings, but I'm confused as to whether templates are needed or not. Everything I've tried so far has resulted in Filebeat ignoring the custom index when it detects that Elasticsearch has ILM enabled, and just putting everything in the filebeat-${beat.version} index that Filebeat creates by default.

Here's what our config currently looks like. I'm wondering how much of this needs to be changed on the Filebeat config side, and how much (if any of it) can be controlled through pod annotations.

apiVersion: v1
kind: ConfigMap
  name: filebeat-config
  namespace: monitoring
    k8s-app: filebeat
  filebeat.yml: |-
    # Using hints-based autodiscover
        - type: kubernetes
          host: ${NODE_NAME}
          # Only collect logs if pods have the annotation "co.elastic.logs/enabled: true".
          hints.enabled: true
          hints.default_config.enabled: false
            type: container
              - /var/log/containers/*${}.log
            exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines

      - add_cloud_metadata:
      - add_host_metadata:
      # Add deployment environment field to every event to make it easier to sort between Dev and SQA logs.
      - add_fields:
          target: ''
            environment: dev

      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME:elastic}
      password: ${ELASTICSEARCH_PASSWORD:changeme}

Thanks in advance!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.