Snapshot restore data stream problems

Elastic Stack Elasticsearch
,
1

Running 8.13

I am working on restoring a data stream from a snapshot. I am doing this in a surgical way as I need to be able to write to indexes in snapshots while the restore of all other indexes happens in the background.

I do all requests via API, I don't use the web interface.
Here is my current process:

  • Restore snapshot with following payload. This just restores the global state and no indexes.
POST _snapshot/our-repo/snapshot-to-use/_restore?pretty
{
  "indices": [
       "-*","-.*"
    ],
    "include_global_state": true
}

When doing this I see a backing index for each data stream is automatically created. Unfortunately, that index does not have the correct data types which causes conflicts if data is coming in.

The workaround is to quickly rollover each data stream, doing so, the new backing index carries the correct data types.

First of all, why is this happening? I can only assume its because the template mappings are not getting applied right after a restore of global state, therefore the index created will carry default data types.

Second of all, is there a way to avoid this? Is there a way to make it so no backing indexes are automatically added to the data stream after restoring of global state? Or is it possible to disable writing to the indexes, then I can rollover, start writing, and delete this auto created indexes.

I have tried the following:

  • Tried the basic recommended way to disable specific settings (Disable GeoIP database downloader and ILM history store, stop ILM, stop ML, stop watcher) did not help.
  • Tried to disable index writing with "index.blocks.write": true, but this seems to disable Kibana from understanding anything is being written to. The backing indexes for the data streams STILL get data written to them, so if I enable the blocks write setting, then all the data just appears in Kibana. The data is still written to the index with the wrong data types.
  • Tried to restore global state and include some data stream indexes. This just does not work; the default backing index still automatically gets created. Also, the indexes I restored never get put in the backing index. Think this happens by design and is a different problem which I have a different work around for.