Best practices around moving documents between indices

Elastic Stack Elasticsearch
1

Hi all,

I have a question about the best approach to move data between indices when the mapping needs to be updated--I'm working with version 5.5. As an example, let's say I have my_index_v1 setup like this:

my_index_v1 {
  "mapping": {
    "my_type": {
      "properties": {
        "field_one": {
           "type": "text"
        },
         "time_stamp": {
            "type": "date"
         }
      }
    }
  }
}

Now, my plan was to create my_index_v2, with a field_one property set to type "keyword", and then reindex my_index_v1 into my_index_v2, so that all of the documents in my_index_v1 would be moved to my_index_v2 with the correct mappings. This however doesn't actually seem to be possible, since when I go to create the new index with the updated mappings, exceptions are thrown on fields with the same name between the two indices (which are also of the same document type).

My question then is, what is the best approach to moving documents from one index to another when the type has to change? For example, would it work to create my_index_v2 with a property field_one_v2 with the desired mapping, and during the reindex process, have everything stored in field_one, inserted into field_one_v2? Or is my only real option to create my new index with properties that have a different name, then use the scroll API to pull down documents from my_index_v1, map the fields returned to the fields in my_index_v2 and then upload them to my_index_v2?

Thanks!

2

There should be no problems with having the same field names defined in the mappings of two document types with the same name across two indexes. The following works fine:

PUT my_index_v1
{
  "mappings": {
    "my_type": {
      "properties": {
        "field_one": {
          "type": "text"
        },
        "time_stamp": {
          "type": "date"
        }
      }
    }
  }
}

PUT my_index_v1/my_type/1
{
  "field_one": "Hello world!"
}

PUT my_index_v2
{
  "mappings": {
    "my_type": {
      "properties": {
        "field_one": {
          "type": "keyword"
        },
        "time_stamp": {
          "type": "date"
        }
      }
    }
  }
}

POST _reindex
{
  "source": {
    "index": "my_index_v1"
  },
  "dest": {
    "index": "my_index_v2"
  }
}

GET my_index_v2/my_type/1

GET my_index_v2/_mapping

What are the exceptions you are getting?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.