Reindex with embeddings

Elastic Stack Elasticsearch
,
1

I was trying to use the E5 model to generate embeddings for non english documents and I created a field that was a sparse_vector. The index where I changed the mapping already
has a sparse_vector for another embedding and it has been populated.

I think I need a dense vector here, but I already created the mapping as sparse_vector.

Initially I tried to delete the new property in the mapping but that is impossible. So I arrived here:

the suggestion is to reindex while at the same time deleting the unwanted property:

POST /_reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
  },
  "script": {
    "inline": "ctx._source.remove('whatever')"
  }
}

I tried running something like that but I got an error saying I was trying to reindex more than 1000 fields. I took a look at the task response and realized that every field created by embedding is now treated as a field to be re-indexed. Basically I want my old index back, including the embeddings I had already generated. When I do a clone the new field is present in the mapping for the new index.

Is there a way for me to copy an index that already contains embeddings into another index while at the same time removing this unwanted new embedding field?

2

Hi @Carlos_Fernando_Palm

Did you try like this?

POST _reindex
{
  "source": {
    "_source": {
      "includes": ["fielda", "fieldb"] // Specify the fields you want to copy
    },
    "index": "idx_source"
  },
  "dest": {
    "index": "idx_dest"
  }
}