Rollover With Snapshot Restore

[Nakshathri (Manjunatha Nakshathri)]

Hello All,

We have time-based data which is getting saved in an index. As a common challenge with such patterns, the index is becoming huge and it's affecting the performance. And also having a single index limits the capability on how easily we can retire old not required data set.

To overcome this, we are thinking of using rollover api to dynamically create indexes based on the number of documents and a day parameter. When we were doing a dry run, we realized that index name to be of a certain pattern ( ending with digits) to enable rollover. Unfortunately, our existing index does not comply with this pattern and hence forced to rename the index before setting up the rollover option.

Option 1

• Take a snapshot of the existing index
• Create a new index template
• Restore the backup to newer index
• Set up the rollover strategy

Setup Instructions used :

-- Take a backup based on the snapshot
curl -XPUT 'http://10.1.2.9:9200/_snapshot/log-backup/log-09-02-2019? 
wait_for_completion=true&pretty=true' -d '{
  "indices": "log",
  "ignore_unavailable": true,
  "include_global_state": false
}'

-- Create a index Template 
curl -XPUT 'http://10.1.2.9:9200/_template/log' -d '
{
  "template": "log-*",
  "settings": {
    "number_of_shards":   5,
    "number_of_replicas": 0
 },
"aliases": {
     "search-log": {}
 }
}'

-- Create the Index based on the active template created earlier; 
curl -XPUT 'http://10.1.2.9:9200/log-1'

-- Restore the snapshot in the new index 
curl -XPOST 'http://10.1.2.9:9200/_snapshot/log-backup/log-09-02- 2019/_restore' 
-d '{
 "include_aliases" : false,
 "ignore_unavailable": "true",
 "include_global_state": false,
 "rename_pattern": "log",
 "rename_replacement": "log-1"
}'

-- create active Alias
curl -XPUT 'http://10.1.2.9:9200/log-1/_alias/active-log'

-- Create a Roll over condition for the vehiclelog indexes; This needs to be called 
from cronjob at a regular interval so that the roll over can happen
curl -XPOST 'http://10.1.2.9:9200/active-log/_rollover' -d '
{
  "conditions": {
    "max_age":   "30d",
    "max_docs":  10000
     }
   }'

Issues faced:

• While we setup the rollover strategy, we get an error that index "log" does not match the required regex pattern.

Option 2

• Create a new index template
• Create the new index based on the previous template
• Setup the rollover strategy
• Reindex the data from existing index to the new index

Would like to know in option-1 we get that error. Also whats the preferred option. Really appreciate any feedback.

Regards,
Manjunath