I have an index with a dense_vector
field with 256 dimensions. I want to change to a new embedding model that uses a different number of dimensions. I dont mind losing the data in the old field. What's the best way to keep the data (other than the dense_vector
) and change the dims?
I've tried:
- Looking into deleting a field -- but this appears to be impossible in Elastic.
- Creating a new index with the new mapping with a higher dimensional dense_vector and then reindexing from old to new. This failed during reindex because the dimensions of that field didn't match up.
- Creating a new index that was missing the dense_vector field, and then reindexing from old to new. This worked for reindexing, but then when I wanted to add the field to the mapping, it failed with
mapper [source_title_vector] cannot be changed from type [float] to [dense_vector]
...probably because the field exists in the data without a mapping (I guess its treated as a float?)
Do I need to write some code to go through the documents one at a time and insert them without this field? Any help would be appreciated.