[SOLVED] Filebeat creates only one index

Elastic Stack Beats
1

Hello,

I'm using the default Filebeat configuration on a Kubernetes cluster. This is my configuration:

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - decode_json_fields:
          fields: ["message"]
          target: "json_message"
          process_array: true
          overwrite_keys: true

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      protocol: https
      ssl:
        certificate_authorities: /etc/ssl/certs/client/tls.crt

It works like a charm, the only issue is that Filebeat does not seem to create daily indices. It only created one, and it kept on using it on subsequent days. As far as I understand, Filebeat is supposed to create one index per day. This is the output of the _cat/shards request:

.kibana_task_manager_1           0 p STARTED        2  21.6kb 10.60.19.39 elasticsearch-logging-es-default-2
.kibana_task_manager_1           0 r STARTED        2  21.6kb 10.60.21.43 elasticsearch-logging-es-default-1
.kibana_1                        0 r STARTED        8 196.4kb 10.60.19.39 elasticsearch-logging-es-default-2
.kibana_1                        0 p STARTED        8 196.4kb 10.60.21.43 elasticsearch-logging-es-default-1
filebeat-7.4.2-2019.12.02-000001 0 r STARTED 32473955  18.6gb 10.60.19.39 elasticsearch-logging-es-default-2
filebeat-7.4.2-2019.12.02-000001 0 p STARTED 32473434  18.8gb 10.60.20.24 elasticsearch-logging-es-default-0
.security-7                      0 p STARTED       36  88.9kb 10.60.20.24 elasticsearch-logging-es-default-0
.security-7                      0 r STARTED       36  88.9kb 10.60.21.43 elasticsearch-logging-es-default-1
.apm-agent-configuration         0 p STARTED        0    283b 10.60.19.39 elasticsearch-logging-es-default-2
.apm-agent-configuration         0 r STARTED        0    283b 10.60.20.24 elasticsearch-logging-es-default-0

As you can see, the only index created is the one that was created when I deployed filebeat, 3 days ago.

I'm lost really. Should I add additional configuration for Filebeat to behave the way I expect it to behave?

Thanks!

2

Alright, this wasn't working because of ILM. I deactivated it and everything is working like a charm now.

Thanks!

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.