ElasticCloud rollover

[mfa]

I have sat up lifecycle policy for index in order to delete the index if it gets over 50 gb. The index is called: myindex-000001 and when the policy applies, it creates the second index which calls it: myindex-000002 automatically.
The application (node js) is pointing to the first index as it is not aware og the new index therefor it doesn't send data to the new created index and as a result it gives error on the first index that shards failed.
I'm wondering how this can be fixed?
Should the application send data to alias of index? or can it be send to myindex-* so that in case of rollover and getting new number it still can send data?

[warkolm (Mark Walkom)]

@mfa

Should the application send data to alias of index?

Ye, you should be using the write alias for the policy as you defined it.

[mfa]

@warkolm

write alias for the policy

can you give me an example please?

This is my policy:

    PUT _ilm/policy/deleting_policy
    {
      "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "25d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
     }
     }
    }

[warkolm (Mark Walkom)]

What does the template that you setup look like?

You need to look for index.lifecycle.rollover_alias as per Rollover | Elasticsearch Guide [7.12] | Elastic

[mfa]

Here is my template:

PUT _component_template/template1
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "ttlInDays": {
          "type": "integer"
        },
        "@log_group": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "@message": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "@owner": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "versionId": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "awsRegion": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "ip": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "sensitiveData": {
          "type": "text"
        },
        "correlationId": {
          "type": "text"
        },
        "serviceId": {
          "type": "text"
        }
      }
    }
  }
}

PUT _index_template/template_1
{
  "index_patterns": ["myindex*", "index2*", "index3*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "index.lifecycle.name": "deleting_policy"
    }
  },
  "priority": 200,
  "composed_of": ["template1"],
  "version": 3,
  "_meta": {
    "description": "Prod Template"
  }
}

I have several indexes that each of them has it's own alias name. How can I add index.lifecycle.rollover_alias in template properly to assign the alias name for each index?

What about the node-js app that points to the first index -00001, not the rollover one ?

[warkolm (Mark Walkom)]

@mfa

I have several indexes that each of them has it's own alias name. How can I add index.lifecycle.rollover_alias in template properly to assign the alias name for each index?

The docs go into this, so it's best to start there and let us know if you have issues.

@mfa

What about the node-js app that points to the first index -00001, not the rollover one ?

You need to update it to use the alias.

[mfa]

@mfa

index.lifecycle.rollover_alias

can you please send me the link to the doc? I have searched a while for this issue but couldn't find a solution to set the index.lifecycle.rollover_alias properly in the template therefor, every time I have to add it manually after index creation.
Someone also mentioned it here in 2019 but it hasn't been fixed or given any solution for that:

ILM should support placeholders in rollover_alias

This is really a big issue and needs a solution for it.

[warkolm (Mark Walkom)]

That issue is not related to what you are doing as far as I can see.

The link is in my first post, but - Rollover | Elasticsearch Guide [7.12] | Elastic

[mfa]

This is actually completely related. Maybe I haven't express it clearly in my first post, sorry for that.
But today I add index.lifecycle.rollover_alias manually after I check and see that a new index has been created using template, and while I don't check it, the error stays there and the teams get problem until someone fix it manually.
If I would be able to add this under the template then it would be created automatically in index creation time.

settings": {
    "index.lifecycle.name": "my_policy",      
    "index.lifecycle.rollover_alias": "{write_index}-alias"    
  }

At the moment the only solution is to create separated template for each index which doesn't seem nice. If someone has over 10 indexes it needs over 10 templates...
Is this issue, that mentioned in that post, under development?

[warkolm (Mark Walkom)]

It doesn't look like that's being worked on, as it's been pushed into datastreams functionality.