Can we use Scann for vector similarity in elasticsearch?

[prakritidev (Prakritidev Verma)]

Hi,

I am using es 7.14 and the cosine similairty function is not optimal as compare to other technologies.

Can I use ScaNN somehow instead ? How will i integrate that library is thats possible.

Thanks.

[BenTrent]

Hey @prakritidev , You cannot use your own approximate nearest neighbor algorithm.

In 8.0, Elasticsearch supports HNSW for larger number of vectors, and in more recent releases we have integrated the support of int8 quantized vectors. Thus saving on space and increasing search time.

[prakritidev (Prakritidev Verma)]

Sure, as far as I know int8 quantized vectors consume less disk. My problem statement is that same vectors that i got from BERT performs better similarity search when I use ScaNN but works very bad when I'm using elasticsearch inbuilt function. Are you implying that int8 quantized vector will improve the quality if consine similary as well ?

PS: I'm not a data scientist.