Not able to change filebeat index name running on Kubernetes

[Vinicius_Maia]

Hello guys,
I'm not able to change my index name in filebeat running on kubernetes.
I found a lot of information about it, but my setup isnt' working. I'll really appreciate for any help.
My objective is send my log files from my kubernetes DEV cluster with filebeat-tst-* index name and my log files from my kubernetes STAGE cluster with filebeat-stg-* to the same Elasticsearch cluster.
With this i'll be able to split the info.
Following my yaml file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: monitoring
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

    setup.template.name: "filebeat-tst"
    setup.template.pattern: "filebeat-*"
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      index: "filebeat-tst-%{[agent.version]}-%{+yyyy.MM.dd}"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: monitoring
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.7.1
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: 10.0.14.10 # Vip Elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
            cpu: 300m
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /u01/data/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /u01/data/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate

[shaunak (Shaunak Kashyap)]

I don't think what you're seeing is specific to running Filebeat in Kubernetes. I suspect this is a general Filebeat issue, regardless of where you run it.

Looking at your Filebeat configuration, I think you are running into the issue where the output.elasticsearch.index setting is being ignored because Filebeat is using index lifecycle management aka ILM (which is the default behavior starting Filebeat 7.0.0 onwards). You can read more about this issue and how to mitigate it at https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#index-option-es.

[Vinicius_Maia]

@Vinicius_Maia

filebeat-tst

Thanks @shaunak I was going to answer my own question. I found the answer some hours ago in the link: https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html

The problem is about the ILM and to figure it out is necessary disable ILM in the filebeat setup.

Following my ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: monitoring
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    setup.ilm.enabled: false
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
    
    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    setup.template.name: "filebeat-stg"
    setup.template.overwrite: true
    setup.template.enabled: true
    setup.template.pattern: "filebeat-stg-*"
    
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      index: "filebeat-stg-%{[agent.version]}-%{+yyyy.MM.dd}"