# Partition HNSW graph per user, elastic KNN
**URL:** https://discuss.elastic.co/t/partition-hnsw-graph-per-user-elastic-knn/346394
**Category:** Elasticsearch
**Tags:** vector-search
**Created:** [November 3, 2023, 5:39pm UTC](https://discuss.elastic.co/t/partition-hnsw-graph-per-user-elastic-knn/346394 "2023-11-03T17:39:53Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@s.ankursonawane](https://discuss.elastic.co/u/s.ankursonawane)
#### Post date: [November 3, 2023, 5:39pm UTC](https://discuss.elastic.co/t/partition-hnsw-graph-per-user-elastic-knn/346394/1 "2023-11-03T17:39:53Z")
Hi, I was exploring the performance of Elastic KNN search scaling. I would like to section off my hnsw graph per user as my searches will always be filtered by the user. The only way I can think to do this in ES seems to be creating separate indices. Is there any other way to do so?
For example in pinecone they have the concept of namespaces, which allow us to partition records based on some key
---
### Author:  [@BenTrent](https://discuss.elastic.co/u/BenTrent)
#### Post date: [November 3, 2023, 6:52pm UTC](https://discuss.elastic.co/t/partition-hnsw-graph-per-user-elastic-knn/346394/2 "2023-11-03T18:52:26Z")
@s.ankursonawane
So, for fully partitioned graphs, you have to do separate indices.
But, may I ask what the usecase is? How many users? How many vectors per user?
If you are only on the scale of 1M vectors of 768 float32 dimensions, you could use brute force "index: false", and sort the index by user ID. This way, we can page in the vectors for a user. This could all be in the same index, partitioned by user and searched quickly.
However, if your scale requires HNSW (10M+), separate indices is the way to go.
---
### Author:  [@s.ankursonawane](https://discuss.elastic.co/u/s.ankursonawane)
#### Post date: [November 10, 2023, 9:43pm UTC](https://discuss.elastic.co/t/partition-hnsw-graph-per-user-elastic-knn/346394/3 "2023-11-10T21:43:05Z")
hi @BenTrent
Number of Users: ~100k
Number of vectors per user: ~1M+
We would really prefer not to use brute force, but I understand that per user indices is an anti pattern which wouldn't scale, right?
---
### Author:  [@s.ankursonawane](https://discuss.elastic.co/u/s.ankursonawane)
#### Post date: [November 16, 2023, 10:00pm UTC](https://discuss.elastic.co/t/partition-hnsw-graph-per-user-elastic-knn/346394/4 "2023-11-16T22:00:41Z")
Hi @BenTrent your input on this would really be helpful
---
### Author:  [@BenTrent](https://discuss.elastic.co/u/BenTrent)
#### Post date: [November 17, 2023, 2:21pm UTC](https://discuss.elastic.co/t/partition-hnsw-graph-per-user-elastic-knn/346394/5 "2023-11-17T14:21:53Z")
@s.ankursonawane ,
Those numbers indicate you will have 10+ billion vectors but only querying \< 0.001% at a time. I still think you should sort your index by user ID and brute force query the vectors. This way all user vectors are stored next to eachother per segment & shard, allowing the search to go directly to the section that contains the user and search. Additionally, your vector dimensions should be smaller if possible, I don't know what models you are considering (\< 384dims or so).
The querying HNSW graphs (or any approximate vector index) and restrictive filtering (in your case filtering by less than \< 0.001%) is a really bad idea.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 15, 2023, 2:21pm UTC](https://discuss.elastic.co/t/partition-hnsw-graph-per-user-elastic-knn/346394/6 "2023-12-15T14:21:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.