Hi everyone, I have performance issues with vector search. Can anyone please help! 
I have the following setup:
- 5 node running on K8s
- Each node has 32vCPU, 128GB RAM
- Total index size ~6.5TB at the moment (roughly 90.000.000 documents)
- Each document includes a dense vector field for performing vector search. The dimension of the field is 768!
After reading the documentation for performance optimisation for kNN search (Tune approximate kNN search | Elasticsearch Guide [8.10] | Elastic), I did the following:
- Make sure the cluster has sufficient memory to hold the HNSW graph in RAM.
- Use the dot product.
- Reduce the number of segments to 2 (per shard).
- Warm up the filesystem cache.
This resulted in desired response times for our use case! 
However…
- When doing heavy inserts into the index the segments increase, and performance drops.
- As a potential solution to this I decided to index all data into a new index, decrease segments, reassign alias, and drop the old index. This to avoid performance degradation during indexing.
- It seems, however, that the heavy indexing pushes the HNSW graph out of the RAM, making searches slow again…
I can see insanely high disk IO when doing kNN searches during these inserts.
Some potential solutions:
- Increase nodes to give more RAM in the hope that the HNSW graph is not being pushed out of RAM.
- Research the possibility to have dedicated nodes that take care of indexing such that the HNSW graph is not pushed out. Not even sure whether this is possible? Anyone ideas?
- Hopefully someone else has another idea?
Thanks in advance!