Partition HNSW graph per user, elastic KNN

[s.ankursonawane (Ankur S)]

Hi, I was exploring the performance of Elastic KNN search scaling. I would like to section off my hnsw graph per user as my searches will always be filtered by the user. The only way I can think to do this in ES seems to be creating separate indices. Is there any other way to do so?

For example in pinecone they have the concept of namespaces, which allow us to partition records based on some key

[BenTrent]

@s.ankursonawane

So, for fully partitioned graphs, you have to do separate indices.

But, may I ask what the usecase is? How many users? How many vectors per user?

If you are only on the scale of 1M vectors of 768 float32 dimensions, you could use brute force "index: false", and sort the index by user ID. This way, we can page in the vectors for a user. This could all be in the same index, partitioned by user and searched quickly.

However, if your scale requires HNSW (10M+), separate indices is the way to go.

[s.ankursonawane (Ankur S)]

hi @BenTrent

Number of Users: ~100k
Number of vectors per user: ~1M+

We would really prefer not to use brute force, but I understand that per user indices is an anti pattern which wouldn't scale, right?

[s.ankursonawane (Ankur S)]

Hi @BenTrent your input on this would really be helpful

[BenTrent]

@s.ankursonawane ,

Those numbers indicate you will have 10+ billion vectors but only querying < 0.001% at a time. I still think you should sort your index by user ID and brute force query the vectors. This way all user vectors are stored next to eachother per segment & shard, allowing the search to go directly to the section that contains the user and search. Additionally, your vector dimensions should be smaller if possible, I don't know what models you are considering (< 384dims or so).

The querying HNSW graphs (or any approximate vector index) and restrictive filtering (in your case filtering by less than < 0.001%) is a really bad idea.