Issue setting up Index Lifecycle policy correctly

[ozonshak]

Hello. I'm new to setting up a proper Lifecycle policy, so I'm hoping someone can please give me a hand with this. So, I have an existing index getting created on a weekly basis. This is a third party integration (they provided me with the pipeline and index template for the incoming logs). Logs are being created weekly in the pattern "name-YYYY-MM-DD". I'm attempting to setup a lifecycle policy for these indexes so they transition from hot->warm->delete. So far, I have done the following:

Updated the index template to add the policy and set an alias:

{
  "index": {
    "lifecycle": {
      "name": "Cloudflare",
      "rollover_alias": "cloudflare"
    },
    "mapping": {
      "ignore_malformed": "true"
    },
    "number_of_shards": "1",
    "number_of_replicas": "1"

On the existing indexes, set the alias and which one is the "write" index:

POST /_aliases
{
    "actions" : [
        {
            "add" : {
                 "index" : "cloudflare-2020-07-13",
                 "alias" : "cloudflare",
                 "is_write_index" : true
            }
        }
    ]
}

POST /_aliases
{
    "actions" : [
        {
            "add" : {
                 "index" : "cloudflare-2020-07-06",
                 "alias" : "cloudflare",
                 "is_write_index" : false
            }
        }
    ]
}

Once I did that, I started seeing the following 2 errors (1 on each index):

I'm not sure why the "is not the write index" error is showing up on the older index. Perhaps this is because it is still "hot" and trying to move it to another phase without it being the write index?

For the second error, is this because the name of the index is wrong for rollover?

I'm also not clear if this is a good scenario for rollover. These indexes are being created weekly, which I assume is ok. I would think normally you would create a single index and let the policy split off the older ones based upon your criteria (size, age, etc). Should I change this or can I make this policy work with existing weekly files? In case you need it, here is part of the pipeline that I imported into ElasticSearch that I believe is responsible for the index naming:

      {
        "date_index_name" : {
          "field" : "EdgeStartTimestamp",
          "index_name_prefix" : "cloudflare-",
          "date_rounding" : "w",
          "timezone" : "UTC",
          "date_formats" : [
            "uuuu-MM-dd'T'HH:mm:ssX",
            "uuuu-MM-dd'T'HH:mm:ss.SSSX",
            "yyyy-MM-dd'T'HH:mm:ssZ",
            "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
          ]
        }
      },

[ozonshak]

So, for the "number_format_exception", I'm thinking it is due to this setting I'm seeing in the index (provided_name):

{
  "settings": {
    "index": {
      "lifecycle": {
        "name": "Cloudflare",
        "rollover_alias": "cloudflare"
      },
      "mapping": {
        "ignore_malformed": "true"
      },
      "number_of_shards": "1",
      "provided_name": "<cloudflare-{2020-07-20||/w{yyyy-MM-dd|UTC}}>",
      "creation_date": "1595203589799",
      "priority": "100",
      "number_of_replicas": "1",

I believe this "provided_name" is getting established from the pipeline's "date_index_name" I provided above. If this is the issue, is there a way to create a fixed index name via the ingest pipeline without it changing based upon the date? I would rather just created a fixed index and let the lifecycle policy handle the split offs (i.e. 0001, 0002, etc).

[ozonshak]

Just a bump here to see if anyone can help with this? Thanks.

[warkolm (Mark Walkom)]

For clarities sake; If you roll daily with ILM you don't end up with the date in the index name after the first index. The date in the index name is going to represent the date that the policy was first invoked.

So;

@ozonshak

I would think normally you would create a single index and let the policy split off the older ones based upon your criteria (size, age, etc). Should I change this or can I make this policy work with existing weekly files?

Yes change it and let ILM handle the split.

@ozonshak

here is part of the pipeline that I imported into Elasticsearch that I believe is responsible for the index naming

That's helpful, thanks! To backup the previous comment, I would suggest removing that from the pipeline, sending the docs to cloudflare as the writable alias, and then let ILM manage the rollover.

[ozonshak]

Thanks @warkolm. So, I tried that just a couple of days ago, but it still isn't working. I basically removed the entire "date_index_name" processor from the pipeline (so it won't create an index with that naming convention including the date) and bootstrapped the first index with the name I want ending in -000001.
I was hoping as logs hit the pipeline and get ingested, the pipeline would dump the logs into the "cloudflare-2020-08-11-000001" index (name I gave the index when I bootstrapped it) and the ILM would manage everything at that point.

Now, I'm getting this error from the code that is sending the logs into the pipeline:

index [cloudflare-*], type [_doc], id [null], message [ElasticsearchException[Elasticsearch exception [type=security_exception, reason=action [indices:admin/create] is unauthorized for user [cf-logs]]]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_state_exception, reason=There are no external requests known to support wildcards that don't support replacing their indices]];]

I basically removed the pipeline, re-created it, and then bootstrapped the initial index. I don't think this is a user privilege issue (that part of the error message might be a red herring). cf-logs is a superuser and I was already using this user previously. Looking at "reason=There are no external requests known to support wildcards that don't support replacing their indices", this might occur when an index isn't specified during an operation. I'm wondering if perhaps the pipeline, no longer having any index naming rules, is unable to create or find an existing index now? Just a guess.

Just for kicks, I tried switching back to my original config and the logs started flowing again. With the "date_index_name" processor, indexes are being created weekly at the moment. So, there doesn't seem to be an issue with the pipeline creating new indexes under this user. There must be something else going on here.

Any idea what is wrong?