We have an ES index with a dense_vector field containing approx. 2.5 M documents which we use for ANN search. The embedding dimension is 512. We experience a very uneven search performance, it often takes >20 s for the searches to complete, however they sometimes complete in less than 1 s. The index is receiving frequent writes, however we have also created a copy of the index which does not receive writes where we have excluded the dense_vector field from the source. That did not seem to improve the performance substantially, however after reading Slow aKNN search we did try to force merge the segments in this read copy, which did in fact give a more even and faster search performance around 0.5 to 1 s when reducing the segments from 20 to 5.
As we do need to keep our production index updated, we do need to be able to receive writes to it. Is there a good way to reduce the number of segments in an index without making it read-only, as we have seen that it is not recommended to force-merge an index that is receiving writes. We currently have 25 segments in our production index, of which some of them are very small. We are running ES version 8.7.0.
Hi,
Assuming that the data is time based, can we use ILM to generate new indexes on a rolling basis every day, and forcemerge all the original ones to improve the retrieval performance of KNN?
Thanks
Hi,
Thanks for your answer and sorry for the late reply. The data is not time based, it is product data where documents can be updated/deleted at any time depending on if it is available in stock or not.
Hi Therese,
Indeed it is not recommended to do force merge on an undateable index. If there is some core data that doesn't change, you can keep it into a single segment index, while changeable parts can be in another index; and do search on an index pattern encompassing all these indices.
Do you know how many segments do you have when searches take 20s vs 1s? Could the degradation also be related to something else (e.g. sudden increase in number of searches, searches requests are waiting in queue etc?).
Hi Mayya,
Great questions, we have not yet been able to figure out why some searches are much faster than others. There is no noticeable difference in the number of segments, we currently have 24 segments and the search speed varies a lot between searches. There is no noticeable difference in search load either. We are using elastic cloud for hosting and one thing we have noticed that will temporarily speed up the searches is if we change the size of the deployment. It works even if we first scale up and then scale down the deployment to the original size, but it is only faster for a while, then it becomes slower again.
First I want to acknowledge that large number of segments slow down searches. In our own benchmarks searches on a force-merged index run 10 times faster than on changing index with many segments. This is a known problem and we intend to work on it.
But it looks like in your case even if you keep segments the same (24) , you still get different search speeds? Is the index being modified?
Another factor that influences search speed is the ability for vectors to be in memory (outside of java heap). If this memory is not enough and it is shared with other processes, vectors have to read from disk which slows down searches.
Finally, as an Elastic customer, you can open a support ticket for us to investigate futher.
Thank you for your response! Do you know what the approximate time frame is for working on fixing this issue?
I do agree with you that there is likely something else affecting our issues as well, since we have a very large difference in search speed. We are continuously writing to the index, however as I mentioned in my original post we did also try to create a copy of the index which was not receiving any writes and we still experienced a large variety in search speeds.
Is there a way to check if all vectors fit in memory outside of the java heap?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
