Aliases and indices on demand

Elastic Stack Elasticsearch
1

We have many indices in our system, and we're following the general guideline to create indices only on demand.
We're also starting to use aliases to allow for rollover and for background data migrations (using the index naming such as <index_base_name>-000001

The problem is that until now an index was automatically created when indexing the first document using a known index name.
However now when we need to use aliases with backing indices, we don't know the index name, and since the index doesn't yet exist and so as the alias, we cannot use the alias for indexing the document.
It seems this means that either:

  1. We need to check before each write if the alias/index exists. If not - create the index (with -000001 suffix) and the alias. If it does exist, just use the alias. This can work with a "try-catch" in the code, but it doesn't seem right to me.
  2. Create the index in advance, which means we will have many empty indices - not a good thing.

What would be the correct way of doing this?

2

Have you considered using ILM ?

3

Yes, we plan to use ILM for rollover policies.
However as far as I understand it comes into effect after the first index already exists (e.g. becomes full or some time interval has passed).
I'm having a problem with the creation of the first index (see original question for the specific scenario).
If ILM can help I would be happy for some more specific guidance.

4

ILM has a write and a read alias, so you seperate those two actions and don't need to worry about the underlying index name.

5

According to this:

When you set up policies for your own rolling indices, you need to manually create the first index managed by a policy and designate it as the write index.

So this connects to my question about creating the initial index.

6

You can setup an index template that will apply to any matching index_base_name indices, and they will have the aliases assigned to them.

7

I'm not sure I understand.
Let's say I setup the alias ('my_alias') in the index template.
In my backend code I have something like which uses the alias to write documents:

client.index({
  index: 'my_alias',
 body: {
   prop: 'joe'
 })

What will happen when this runs when the index does not yet exist?
Will the index be automatically created? If it will, what will be the index name?

8

No worries!

If you don't have an index, then setup an ILM policy (with index template) now. That way when you do write to the alias it'll create the backing index automagically.

That depends on what you set your policy to do. It could be a pattern like my-aliased-index-<counter>, or whatever you want.

9

I can't get it to work.
I tried defining the following template and policy, but after that when I try to insert a document using the alias my_alias it just creates an index named my_alias.
Any idea?

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {}
      }
    }
  }
}

PUT _index_template/ppp_template
{
 "index_patterns": ["ppp*"], 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "my_policy", 
      "index.lifecycle.rollover_alias": "my_alias" 
    },
    "aliases": {
      "my_alias": {}
    }
  }  
}
10

Is would be nice to have a working example.
This totally blocks us as we cannot use the combination of aliases + indices creation on-demand.
The docs are not very clear regarding this.

11

Any assistance would be appreciated, we are blocked.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.