Filebeat not creating index daily

[daniel_fablius]

Hi, so i use filebeat to send logs to Elasticsearch
and i see an index is created with name "filebeat-7.7.1-2020.07.29-000002"

i assume that filebeat will keep creating a new index based on the datetime of the data event is sent.
but on 30th July 2020, no new index is created.

is what i assume is wrong? is filebeat not supposed to create a new index everyday?
if what i assume is right, what i should do to make it happened?

[fgjensen (Flemming G. Jensen)]

Hi @daniel_fablius
Could you share your output configuration, please?

[daniel_fablius]

Hi @fgjensen

i will show you my configMap yaml file instead

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: filebeat-config
      namespace: logging
      labels:
        app: filebeat
    data:
      filebeat.yml: |-
        filebeat.config:
          modules:
            path: ${path.config}/modules.d/*.yml
            reload.enabled: true  
        filebeat.autodiscover:
          providers:
            - type: kubernetes
              node: ${NODE_NAME}
              hints.enabled: true
              hints.default_config:
                type: container
                paths:
                  - /var/log/containers/*${data.kubernetes.container.id}.log
        output.elasticsearch:
          hosts: '${ELASTICSEARCH_URL}'
        monitoring:
          enabled: true

i think the one that matters is the one ?

    output.elasticsearch:
      hosts: '${ELASTICSEARCH_URL}'

[fgjensen (Flemming G. Jensen)]

See you are deploying on a Kubernetes cluster You could try to add an index filed to the output like this index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}", see the documentation here: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html. Depending on your use case, you should also consider if you want a new index created every day. The recommended index size is around 50GB

[daniel_fablius]

Yes, im deploying on Kubernetes cluster, my bad to not point it out in the beginning.

so, i try to add the index field, and addition to that i add setup.template.name and setup.template.pattern also (based on this docs https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#index-option-es)

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: filebeat-config
      namespace: logging
      labels:
        app: filebeat
    data:
      filebeat.yml: |-
        filebeat.config:
          modules:
            path: ${path.config}/modules.d/*.yml
            reload.enabled: true  
        filebeat.autodiscover:
          providers:
            - type: kubernetes
              node: ${NODE_NAME}
              hints.enabled: true
              hints.default_config:
                type: container
                paths:
                  - /var/log/containers/*${data.kubernetes.container.id}.log
        output.elasticsearch:
          hosts: '${ELASTICSEARCH_URL}'
        setup.template.name: "filebeat"
        setup.template.pattern: "filebeat-*"  
        monitoring:
          enabled: true

but after i re deploying my cluster, i still not seeing my 30th July (today) index..

*ignore the face that it has 28 and 29 July, its created by ILM

you mentioning about 50GB index size, is ILM what you means? if thats what you mean, i did try that, but since my cluster only live for office hours, the lifetime of an index is not 24 hours, so when i try ILM with 1 day Max age, it wont generate new index in the next day (since the cluster age is not 24 hours yet).

so, is filebeat creating new index based on ILM only, or its creating new index by output configuration? since if its ILM, then i dont have any way to create it every dat then

[daniel_fablius]

thanks for pointing out the docs @fgjensen, turns out i missed out that the ILM enabled by default, and it will disable the index settings in the output config (https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#index-option-es)

so i just need to disable it, and the index for today is created

my last output config

        output.elasticsearch:
          hosts: '${ELASTICSEARCH_URL}'
          index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
        setup.template.name: "filebeat"
        setup.template.pattern: "filebeat-*"  
        setup.ilm.enabled: false

the index and setup.template* have default value, so i suppose those actually not needed to be defined for me (since i use default value anyway)

Thanks for help