The proper way to reindex a data stream index

Elastic Stack Elasticsearch
,
1

I have run into an issue with mappings in an data stream index. Can someone help me out? I am getting data type conflicts between the new mapping and old mapping of some fields. What I want to do is just remove the mappings from the older index in the data stream, but apparently that's not possible. It sounds like I have two options,

  1. Delete the index (I can't do that it has data we need)
  2. Reindex the data stream index.

Now I understand reindexing a simple static index, reindexing an index in a DataStream seems to be different process. I tested in my lab reindexing an index, then created an alias for it to the older index, and deleting the older index. Now when I query against the new index or its alias, I find the data. But if I query against the data stream name, I do not find the data. It seems there is no way to "pop" the new index or its alias back into the data stream.
I have read online that I could re-index the whole data stream, but that is not practical, there's millions and millions of records, it would take forever.
What other options do I have here?

2

I think you have to create "date view".

3

UPDATE: I now have the correct process. This is a real pain to do, and probably should script this. But sharing the current process below. Note: we reindexed to fix wrong data types. The main parts are this

  • First fix the component template mapping to what you want in elastic.
  • Rollover the index, and confirm new data is coming in with the correct mappings.
  • Before reindexing, we needed to create a new _index_template taht did not include enrichment pipelines, and set mappings.properties.@timestamp to ignore_malformed: false Then include the same settings and mappings. The index pattern should match your new pattern you will use. For us its .ds-logs-testing-default-*-reindexed-v2
  • Then you must create each new index, this will build an empty index with the correct mappings.
  • Then you can reindex the old index to the new index created with the new mappings
  • Check count and failure of new indexes
  • Replace data stream backline with new reindexed reindexes
  • Delete old indexes when ready.

Here are the full details. I hope this help someone else because this is not straight forward the first time you do it. The example includes two indexes that are reindexed, you can add or remove as much as you like. Also, yes you could combine all bad indexes into one big index, but I don't like that, that's why I separate them into separate reindexed indexes.

1. Prepare

Find the index template first "logs-testing-template"

GET _index_template/logs-testing-*

Uses logs-testing-settings and logs-testing-mappings

GET _component_template/logs-testing-settings
GET _component_template/logs-testing-mappings

SAVE - Get a list of all indices in the data stream

GET _data_stream/logs-testing-default?human&filter_path=data_streams.name,data_streams.template,data_streams.generation,data_streams.indices.index_name

SAVE - Check mappings of a couple of them before and after the problem

GET .ds-logs-testing-default-2026.06.24-000415/_mapping
GET .ds-logs-testing-default-2026.06.04-000373/_mapping

Gather mappings of all

GET .ds-logs-testing-default-*/_mapping/field/policy_hit?expand_wildcards=all

SAVE - Get count of all documents in the data stream and a couple of indexes

GET logs-testing-default/_count
GET .ds-logs-testing-default-2026.06.24-000415/_count
GET .ds-logs-testing-default-2026.06.04-000373/_count

2. Change mappings of current component template

PUT _component_template/logs-testing-mappings
{
  "version": 2,
  "template": {
    "mappings": {
      "properties": {
        "field01": {
          "type": "integer"
        },
        "field02": {
          "type": "integer"
        },
        "field03": {
          "type": "keyword"
        },
        "field04": {
          "type": "integer"
        },
        "test": {
          "type": "keyword"
        },
        "field05": {
          "type": "integer"
        },
        "field06": {
          "type": "integer"
        },
        "field07": {
          "type": "boolean"
        },
        "field08": {
          "type": "ip"
        },
        "field09": {
          "type": "keyword"
        },
        "@timestamp": {
          "type": "date"
        },
        "field10": {
          "type": "keyword"
        },
        "data_stream": {
          "properties": {
            "namespace": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword"
            },
            "dataset": {
              "type": "keyword"
            }
          }
        },
        "field11": {
          "type": "keyword"
        },
        "field12": {
          "type": "integer"
        },
        "field13": {
          "type": "ip"
        },
		"field14": {
            "type": "ip"
        },
        "event": {
          "properties": {
            "original": {
              "type": "text",
              "index": false
            }
          }
        }
      }
    }
  }
}

Simulate to confirm it has the correct mappings

POST _index_template/_simulate_index/logs-testing-default?filter_path=template.mappings,template.settings,overlapping

Rollover index

POST logs-testing-default/_rollover

Check status of data stream

GET _data_stream/logs-testing-default?human&filter_path=data_streams.name,data_streams.generation,data_streams.template,data_streams.indices.index_name

3. Reindex

Reindex all the current mapped indexes

Create temp template first. Notice how the enrich pipeline was removed. Also adjust the @timestamp to ignore_malformed: false

PUT _index_template/logs-testing-reindexed-v2-template
{
  "index_patterns": [".ds-logs-testing-default-*-reindexed-v2"],
  "template": {
    "settings": {
      "index": {
        "hidden": true
      }
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date", "ignore_malformed": false }
      }
    }
  },
  "composed_of": [
    "logs-testing-settings",
    "logs-testing-mappings"
  ],
  "priority": 600,
  "version": 2
}

Do a simulate to check to make sure data types apply

POST _index_template/_simulate_index/logs-testing-reindexed-v2-template?filter_path=template.mappings,template.settings,overlapping

now create the indexes with the same name matching the new temp index, this will create the correct mappings

PUT /.ds-logs-testing-default-2026.06.05-000375-reindexed-v2
PUT /.ds-logs-testing-default-2026.06.05-000376-reindexed-v2

Check the mappings of a few

GET .ds-logs-testing-default-2026.06.05-000375/_mapping
GET .ds-logs-testing-default-2026.06.05-000375-reindexed-v2/_mapping

Now reindex all of the old inexes to the new ones created (Run all these POSTS together)

POST _reindex?wait_for_completion=false&slices=auto&requests_per_second=5000
{
  "source": {
    "index": ".ds-logs-testing-default-2026.06.05-000375"
  },
  "dest": {
    "index": ".ds-logs-testing-default-2026.06.05-000375-reindexed-v2",
    "op_type": "create",
    "pipeline": "_none"
  }
}

POST _reindex?wait_for_completion=false&slices=auto&requests_per_second=5000
{
  "source": {
    "index": ".ds-logs-testing-default-2026.06.05-000375"
  },
  "dest": {
    "index": ".ds-logs-testing-default-2026.06.05-000375-reindexed-v2",
    "op_type": "create",
    "pipeline": "_none"
  }
}

Check the counts and failures for all of them, if all counts same and no failures, good to swap

GET /.ds-logs-testing-default-2026.06.05-000375-reindexed-v2/_count
GET /.ds-logs-testing-default-2026.06.05-000375/_count
GET /.ds-logs-testing-default-2026.06.05-000376-reindexed-v2/_count
GET /.ds-logs-testing-default-2026.06.05-000376/_count

4. Cleanup, remove old indexes

Remove old backing indexes and replace with new reindexed ones

POST _data_stream/_modify
{
  "actions": [
    {
      "add_backing_index": {
        "data_stream": "logs-testing-default",
        "index": ".ds-logs-testing-default-2026.06.05-000375-reindexed-v2"
      }
    },
    {
      "remove_backing_index": {
        "data_stream": "logs-testing-default",
        "index": ".ds-logs-testing-default-2026.06.05-000375"
      }
    },
    {
      "add_backing_index": {
        "data_stream": "logs-testing-default",
        "index": ".ds-logs-testing-default-2026.06.05-000376-reindexed-v2"
      }
    },
    {
      "remove_backing_index": {
        "data_stream": "logs-testing-default",
        "index": ".ds-logs-testing-default-2026.06.05-000376"
      }
    }
  ]
}

Validate the backing indexes are updated

GET _data_stream/logs-testing-default?human&filter_path=data_streams.name,data_streams.generation,data_streams.indices.index_name

WAIT 2 weeks, then Delete old backing indexes with this

DELETE .ds-logs-testing-default-2026.06.05-000375,.ds-logs-testing-default-2026.06.05-000376