Hi,
The indices of my ES cluster are structured as follows:
deviceA-2018 (size 5gb)
deviceA-2019 (size 1gb)
deviceB-2019 (size 100gb)
The deviceIDs are dynamic and unknown a priori and the indices sizes vary too much between them.
For automatization of index creation I am using "index templates" (e.g. set number of shards). This mechanism is very convenient because the indexation is made via Apache Storm and I do not need to take care of the existance of the index; if it does not exists, it is created with the template settings and if exists, the data is indexed directly.
PUT _template/device_template
{
"index_patterns": ["*-*"],
"settings": {
"number_of_shards": 5
},
...
}
For performance reasons I am restricting the max size of an index using "rollover" functionality.
PUT _ilm/policy/device_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "30GB"
}
}
}
}
}
}
PUT _template/deviceA_template
{
"index_patterns": ["deviceA-2019-*"],
"settings": {
"number_of_shards": 5
},
"settings": {
"index.lifecycle.name": "device_policy",
"index.lifecycle.rollover_alias": "deviceA-2019"
}
...
}
PUT deviceA-2019-000001
{
"aliases": {
"deviceA-2019": {
"is_write_index": true
}
}
}
My problem is that I cannot use these two features in conjuction in an automatic way. The configuration that I tried to use but does not work is:
PUT _template/device_template
{
"index_patterns": ["*-*-000001"],
"settings": {
"number_of_shards": 5
},
"settings": {
"index.lifecycle.name": "device_policy",
"index.lifecycle.rollover_alias": "{indexWITHOUTrolloverID}" //e.g. deviceA-2019
},
"aliases": {
"{indexWITHOUTrolloverID}": { //e.g. deviceA-2019
"is_write_index": true
}
}
...
}
PUT _template/device_template
{
"index_patterns": ["*-*-!00001"],
"settings": {
"number_of_shards": 5
},
"settings": {
"index.lifecycle.name": "device_policy",
"index.lifecycle.rollover_alias": "{indexWITHOUTrolloverID}" //e.g. deviceA-2019
},
...
}
The problem that appears in this issue is very similar to my problem (https://github.com/elastic/kibana/issues/27967). Notice that this issue appears in kibana repo but it it is not related with kibana but with elasticsearch.
Is there any way to do delegate the creation of a index on demand by ElasticSearch with rollover enabled?
Thank you for your help!