Rollover_alias errors on logstash daily indices from client application (fluentd)

[cotjoey (Joey)]

This is the error I am getting on the indices that fluentd is sending directly to Elasticsearch:

illegal_argument_exception: index.lifecycle.rollover_alias [logstash] does not point to index [logstash-2021.02.09]

The indices are daily and have the format "logstash-YYYY.MM.DD" (i.e. logstash-2021.03.17), so I end up with one index per day, i.e.:

logstash-2021.03.17
logstash-2021.03.16
logstash-2021.03.15
logstash-2021.03.14
logstash-2021.03.13
logstash-2021.03.12
[...]

I am wondering how I can set the "logstash" alias to rollover the daily index if it matches the ILM policy rules AND also apply automatically the logstash alias on the new daily indices automatically.

For example, if I set my "logstash-policy" ILM to rollover when the daily index reaches 1GB in size, I'd like it to rotate to "logstash-2021.03.17-000002" for today, as an example. I would also lie to have tomorrow's index "logstash-2021.03.18" to automatically get the "logstash" alias attached to it when it gets created and also rollover automatically as per the "Logstash-policy" ILM.

Can you help me with steps to modify my infrastructure to set those automatically.

Thank you.

[Christian_Dahlqvist]

If you are using rollover I think it is best to not have date in the underlying index names and just rely on the generated sequence numbers as ILM does not need date in the underlying index names to work. I would recommend setting up a new rollover index with a write alias of e.g. logstash. You then change your fluentd config (I assume this must be possible) to write to the logstash alias instead of generating index names with date in them.

[cotjoey (Joey)]

I was able to make it work with a date in the index name, but by making the following modifications to my fluentd Kubernetes yaml file:

- name: FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT
  value: "false"
- name: FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME
  value: "logstash-fluentd-kubernetes"
- name: FLUENT_ELASTICSEARCH_ENABLE_ILM
  value: "true"
- name: FLUENT_ELASTICSEARCH_TEMPLATE_FILE
  value: /host/index_template.json
- name: FLUENT_ELASTICSEARCH_TEMPLATE_NAME
  value: "logstash-fluentd-kubernetes"
- name: FLUENT_ELASTICSEARCH_INCLUDE_TIMESTAMP
  value: "true"
- name: FLUENT_ELASTICSEARCH_APPLICATION_NAME
  value: "logging"

volumes:
- name: es-template
  configMap:
    name: es-template

volumeMounts:
- name: es-template
  mountPath: /host
  readOnly: true

---
apiVersion: v1
data:
  index_template.json: |-
    {
        "index_patterns": [
            "logstash-fluentd-kubernetes-*"
        ]
    }
kind: ConfigMap
metadata:
  name: es-template
  namespace: kube-system

The index now look like this when I force a rollover:

logstash-fluentd-kubernetes-logging-2021.04.06-000002
logstash-fluentd-kubernetes-logging-2021.04.06-000001

... and it keeps writing to the same date it initially created the index with (not creating a new one everyday).

The index template name is: "logstash-fluentd-kubernetes" and its pattern: "logstash-fluentd-kubernetes-*"

[cotjoey (Joey)]

I would like to add that most of my other issues trying to create an index name that started with fluentd-* were caused by not having permission to create that new index with the FLUENT_ELASTICSEARCH_USER user. Check that first.