Where is the KNN index stored, in RAM or disk memory ?
I came across that I used to use ChromaDB to store vectors, I had about 1 million vectors and it stores them all in RAM, it took about 24gb. That's too much, and I'm hoping elastik stores the vectors and index on disk, and still have a fast search.
Where does elastiс store the vector index for fast search ?
Wouldn't it be a problem that as the number of vectors grows, the RAM consumption will increase a lot ?
Have a look at the docs around estimating resource usage for knn.
Do you think absolutely all vectors are stored in RAM, or does the elastiс somehow distribute the data partly on disk, partly on RAM ?
I believe the link I provided states that the data is stored on disk, but that it for performance reasons is important that it fits in the operating system page cache (not heap). As the size of data grows it will at some point exceed what can be kept in the page cache and performance is likely to degrade as disk I/O becomes a limiting factor.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.