What is the best way to change the dims for a dense_vector in an index?

Elastic Stack Elasticsearch
1

I have an index with a dense_vector field with 256 dimensions. I want to change to a new embedding model that uses a different number of dimensions. I dont mind losing the data in the old field. What's the best way to keep the data (other than the dense_vector) and change the dims?

I've tried:

  1. Looking into deleting a field -- but this appears to be impossible in Elastic.
  2. Creating a new index with the new mapping with a higher dimensional dense_vector and then reindexing from old to new. This failed during reindex because the dimensions of that field didn't match up.
  3. Creating a new index that was missing the dense_vector field, and then reindexing from old to new. This worked for reindexing, but then when I wanted to add the field to the mapping, it failed with mapper [source_title_vector] cannot be changed from type [float] to [dense_vector]...probably because the field exists in the data without a mapping (I guess its treated as a float?)

Do I need to write some code to go through the documents one at a time and insert them without this field? Any help would be appreciated.

2

Welcome to our community! :smiley:

You cannot change a mapping type once it has been created in the index. To do that you will need to create a new index with the mapping you want, then reindex into it, which is option 2 above.

I don't know much about dense vectors, but perhaps if you can share your reindex request, mappings and the error we can assist further