Filebeat creates wrong index name

Elastic Stack Beats
,
1

I'm using filebeat and elasticsearch 7.5.0, installed via Elastic's helm charts.

I'm trying to get ES and Filebeat set up in such a way that I can stand up a new cluster inside kubernetes and have everything Just Work. I'm running into an issue where I'm setting the ILM policy via filebeat config with the default index pattern of filebeat-7.5.0-%{now/d}-000001 but instead the index gets the name filebeat-7.5.0 and that seems to ruin the ILM setup, as the index template doesn't match (it's looking for filebeat-7.5.0-*) and thus the ILM policy doesn't get applied to the index. It would also probably have issues rolling over as the write alias has the same name as a concrete index.

Index Template and ILM policy get created correctly, as far as I can tell.

index-management logs from filebeat startup:

2019-12-19T09:10:58.100Z        INFO    [index-management]      idxmgmt/std.go:256      Auto ILM enable success.
2019-12-19T09:10:58.239Z        INFO    [index-management]      idxmgmt/std.go:269      ILM policy successfully loaded.
2019-12-19T09:10:58.240Z        INFO    [index-management]      idxmgmt/std.go:408      Set setup.template.name to '{filebeat-7.5.0 %{now/d}-000001}' as ILM is enabled.
2019-12-19T09:10:58.240Z        INFO    [index-management]      idxmgmt/std.go:413      Set setup.template.pattern to 'filebeat-7.5.0-*' as ILM is enabled.
2019-12-19T09:10:58.240Z        INFO    [index-management]      idxmgmt/std.go:447      Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.5.0 %{now/d}-000001} as ILM is enabled.
2019-12-19T09:10:58.240Z        INFO    [index-management]      idxmgmt/std.go:451      Set settings.index.lifecycle.name in template to {filebeat-7.5.0 {"policy":{"phases":{"delete":{"actions":{"delete":{}},"min_age":"6d"},"hot":{"
actions":{"rollover":{"max_age":"1d","max_size":"1gb"},"set_priority":{"priority":100}},"min_age":"0ms"}}}}} as ILM is enabled.  
2019-12-19T09:10:58.312Z        INFO    template/load.go:89     Template filebeat-7.5.0 already exists and will not be overwritten.
2019-12-19T09:10:58.312Z        INFO    [index-management]      idxmgmt/std.go:293      Loaded index template.
2019-12-19T09:10:58.312Z        INFO    [index-management]      idxmgmt/std.go:304      Write alias successfully generated.

Filebeat config:

filebeat.inputs:
  ...
setup.ilm:
      enabled: true
      policy_file: ilm_policy.json
      policy_name: filebeat-%{[agent.version]}
      pattern: "%{now/d}-000001"
      check_exists: false
      overwrite: true

    output.elasticsearch:
      host: '${NODE_NAME}'
      hosts: '${ELASTICSEARCH_HOSTS:es-master:9200}'

ilm_policy.json:

    {
      "policy" : {
        "phases" : {
          "hot" : {
            "min_age" : "0ms",
            "actions" : {
              "rollover" : {
                "max_size" : "1gb",
                "max_age" : "1d"
              },
              "set_priority" : {
                "priority" : 100
              }
            }
          },
          "delete" : {
            "min_age" : "6d",
            "actions" : {
              "delete" : { }
            }
          }
        }
      }
    }

Any ideas what I'm doing wrong?

2

On a whim I changed check_exists: true and overwrite: false and recreated the cluster, and now the index is created correctly. Is this the intended behavior?

I wanted it to always overwrite the policy so that the ILM policy would be updated if I updated the filebeat config (and could then keep all config in version control rather than have the cluster be a snowflake).

3

Hi @bsundsrud and welcome to discuss :slight_smile:

These are the default values for these settings, it should be fine for many cases.

It is generally a good idea to keep overwrite disabled in Filebeats running in your machines, so if any of them is restarted with an incorrect configuration it doesn't modify the configuration for all your deployment.

If you want to modify the configuration you can do it:

4

Yeah, I'm aware of those methods, I just wanted something declarative rather than interactive. Also since this is running as DaemonSet pods in kubernetes, the second method is unavailable. My next stop is probably something that uses the ES API to check certain settings and either overwrite them or at least alert that they've deviated.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.