How to change the index name of beats logs in kubernetes

Elastic Stack Beats
1

I have beats linked to elastic cloud elasticsearch 7.0.0. We have filebeats and metricbeats successfully sending logs to elasticsearch as both deployment and daemonset in kubernetes environment. However, I am unable to change the index name of the logs using following settings. Filebeat indexes to original filebeat-7.0.0-date and Metricbeat indexes to metricbeat-7.0.0-date. I am only providing the filebeat configmap but my metricbeat configmap is pretty much the same. (regarding index) What am I doing wrong?

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        # Mounted `filebeat-inputs` configmap:
        path: ${path.config}/inputs.d/*.yml
        # Reload inputs configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false
    # To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      hints.enabled: true
    processors:
      - add_cloud_metadata:
      - drop_event.when:
          or:
            - equals:
                kubernetes.namespace: "monitoring"
            - equals:
                kubernetes.namespace: "kube-system"
            - equals:
                kubernetes.namespace: "kube-logging"
            - equals:
                kubernetes.container.name: "web-front"
      - decode_json_fields:
          fields: ["message"]
          process_array: true
          max_depth: 10
          target: ""
          overwrite_keys: false
    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      index: "${LOG_PREFIX}.dbmaker.filebeat-%{+yyyy.MM.dd}"
    setup.template.name: "${LOG_PREFIX}.dbmaker.filebeat"
    setup.template.pattern: "${LOG_PREFIX}.dbmaker.filebeat*"
    setup.kibana:
      host: ${KIBANA_HOST}
    setup.dashboards.enabled: true
2

Could you share the debug logs when filebeat is started? My first guess would be something around indentation of related to the 7.0 changes which introduced ILM.

What templates do you see loaded in Elasticsearch?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.