Trying to create a working ILM

Elastic Stack Elasticsearch
,
1

Hi all,

I hope you can help me solve what I expect is a trivial issue but that I seem unable to solve.

I hve a logstash output configured as:

elasticsearch {
  hosts => ["https://elasticsearch-es-http.mynamespace.svc:9200"]
  ssl_enabled => true
  ssl_certificate_authorities => '/certs/ca.crt'
  ilm_rollover_alias => "test1"
  ilm_pattern => "000001"
  ilm_policy => "test1-policy"
  user => kafka
  password => mysecretpassword
        }

as for my elasticsearch config, I have the following:

PUT _ilm/policy/test1-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_docs": "10000",
            "max_primary_shard_size": "200mb"
            /*"max_age": "1d"*/
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

PUT _index_template/test1-template
{
  "index_patterns": ["test1*"],
  "template": {
    "aliases": {
      "test1": {
        "is_write_index": true
      }
    },
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index": {
        "lifecycle.name": "test1-policy",
        "lifecycle.rollover_alias": "test1"
      }
    }
  }
}

and the role I give the logstash user is:

POST /_security/role/logstash-write
{
  "cluster": [
    "manage_ilm"
  ],
  "indices": [
    {
      "names": [
        "gold-*",
        "gold"
      ],
      "privileges": [
        "write",
        "create",
        "create_index",
        "manage",
        "manage_ilm",
        "all",
        "delete",
        "index",
        "create_doc"
        ]
    }
  ]
}

The proble I have is that however I try to change things, logstash seems to create an index named test1 rather than test1-000001

I have also gotten the following error when trying to fiddle with the index patern.
index name [test1] does not match pattern '^.*-\\d+$'

All I am tring to do is have the lifcycle policy create indexes of format test1-00000X with an alias test1 that represents all indices test1-*

I can't see what I am doing wrong. Can you help?

Thanks.

2

With a lot of messing about, we got it working with the following setup. I also relise I might have sent some incorect info like the indices name in logstash-writer role.

In any case, this is what we did to get it to work. Hope it can help someone.

PUT _ilm/policy/foo-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
              "max_age": "1d",
              "max_primary_shard_size": "20gb"
          }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

PUT _index_template/foo-logs
{
  "index_patterns": ["foo-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.lifecycle.name": "foo-logs-policy",
      "index.lifecycle.rollover_alias": "foo-logs"
    }
  }
}

PUT foo-logs-000001
{
  "aliases": {
    "foo-logs": {
      "is_write_index": true
    }
  }
}