Sparse vector embeddings

Elastic Stack Elasticsearch
1

Hi,
Is there any plan to re-introduce sparse vector fields? It was depreciated and there is this discussion about it.

I would like to use sparse vectors to search by sparse vector embeddings.

For example, I would like to use models such as BGE-M3 to index the lexical scores.

3

ELSER is Sparse Vector or more accurately text expansion.

4

Ok, thanks. I read about it before but was not clear to me what exactly is and why is dependent of term expansion.
Can I use it locally? I just want a field to index my a sparse vectors (I will calculate the vectors with a finetuned version of BGE-M3) and retrieve the documents given a query vector (with a similarity score given by the sum of the product of the weights).

5

Sounds like you want to BYOM and create / load your own embeddings which should be doable.

Perhaps look at Elastic Search Labs for some examples

You can load your model into Elasticsearch as well assuming it meets the requirements

6

Sidenote: sparse vector was added back again for ELSER (see https://github.com/elastic/elasticsearch/pull/98996) nd will probably diverge from rank feature over time.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.