Creating indexes and ILM based on Kubernetes namespace using hints-based autodiscover

[Michael_Davis]

Hi all,

We're just getting started with Filebeat on Kubernetes and the Elastic Stack in general. I'm really impressed with the capabilities of Filebeat so far. The hints-based autodiscover feature is a big deal for us as it allows individual app teams to set their own match patterns and other Filebeat configuration settings based on annotations they place on their pods.

What I can't figure out, though, is how we can tell Filebeat (and by extension Elasticsearch) to create separate indexes based on Kubernetes namespaces when using hints/annotations. At the same time, we also want ILM to function for each index. I've played around with conditions and setup.ilm.x settings, but I'm confused as to whether templates are needed or not. Everything I've tried so far has resulted in Filebeat ignoring the custom index when it detects that Elasticsearch has ILM enabled, and just putting everything in the filebeat-${beat.version} index that Filebeat creates by default.

Here's what our config currently looks like. I'm wondering how much of this needs to be changed on the Filebeat config side, and how much (if any of it) can be controlled through pod annotations.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: monitoring
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # Using hints-based autodiscover
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          # Only collect logs if pods have the annotation "co.elastic.logs/enabled: true".
          hints.enabled: true
          hints.default_config.enabled: false
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines

    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      # Add deployment environment field to every event to make it easier to sort between Dev and SQA logs.
      - add_fields:
          target: ''
          fields:
            environment: dev

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME:elastic}
      password: ${ELASTICSEARCH_PASSWORD:changeme}

Thanks in advance!