How to bootstrap an index template with a part of the name not defined yet?

[ElasticLiver (Dense Man)]

Hi, depending on the hostgroup field, I want to add certain string to part of the index name:

filter{
        if [hostgroup] =~ /mass/ {

                mutate { add_field => { "indexname" => "mass" }}
        }
}
output{
    elasticsearch { 
        hosts => ["111.111.111.111"] 
        index => "myprefix-%{indexname}-000001" 
    }
}

But how I can bootstrap the index template, if I dont have part of the name yet?

PUT myprefix-?????-000001 
{
  "aliases": {
    "myprefix-index": {
      "is_write_index": true
    }
  }
}

[Luca_Belluccini]

Hello @ElasticLiver

Unfortunately the problem with the Rollover API / ILM index bootstrapping is you need to know in advance the all the possible names of the "data flows" you will be receiving in the cluster, so that a write rollover alias is ready on Elasticsearch side.

This is detailed in this Github issue and Elasticsearch is evolving to make "unknown" indices auto-bootstrap the indices with 2 concepts:

• Index Templates v2
• Data Streams

If you know all the possible values, you can bootstrap the indices in advance using:

PUT myprefix-mass-000001 
{
  "aliases": {
    "myprefix-mass": {
      "is_write_index": true
    }
  }
}
PUT myprefix-another-000001 
{
  "aliases": {
    "myprefix-another": {
      "is_write_index": true
    }
  }
}

As you have a dynamic variable, you have to use the manual bootstrapping of the indices and Logstash must be configured as follows:

    elasticsearch { 
        hosts => ["111.111.111.111"]
        ilm_enabled => false
        index => "myprefix-%{indexname}" 
    }

You will need to put in place an index template to associated the indices myprefix-mass-* and myprefix-another-* the rollover_alias respectively of myprefix-mass and myprefix-another and a ILM policy.

Those steps are detailed here.

[Badger]

I'm guessing you could use an http filter to do the put of the template, but that strikes me as something that could go very wrong if you have noise in your input.

If your dataset really does only have a few values for indexname then it would be OK, but every additional value adds cost.

[ElasticLiver (Dense Man)]

Thanks Luca_Belluccini , thanks Badger so, the easyest way to solve this will be using myprefix* index template and then adding a ILM after the index is created?

[Luca_Belluccini]

Each write alias (one per prefix) must have its own index template to set the rollover_alias.
E.g.

PUT _template/myindex_mass_template
{
  "index_patterns": ["myprefix-mass-*"],                 
  "settings": {
    "index.lifecycle.name": "the_policy",      
    "index.lifecycle.rollover_alias": "myprefix-mass"    
  }
}
PUT _template/myindex_another_template
{
  "index_patterns": ["myprefix-another-*"],                 
  "settings": {
    "index.lifecycle.name": "the_policy",      
    "index.lifecycle.rollover_alias": "myprefix-another"    
  }
}

The policy name can be the same or different for both.
Other common settings across both indices can be set using another template which matches both. E.g.

PUT _template/myindex_template
{
  "index_patterns": ["myprefix-*"],                 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  }
}