# Populate a dense vector field for only a subset of the documents **URL:** \ **Category:** Elasticsearch\ **Tags:** vector-search\ **Created:** [August 8, 2023, 6:36am UTC](https://discuss.elastic.co/t/populate-a-dense-vector-field-for-only-a-subset-of-the-documents/340326 "2023-08-08T06:36:09Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![sbruinsje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sbruinsje/32/108568_2.png) [@sbruinsje](https://discuss.elastic.co/u/sbruinsje)\ **Post date:** [August 8, 2023, 6:36am UTC](https://discuss.elastic.co/t/populate-a-dense-vector-field-for-only-a-subset-of-the-documents/340326/1 "2023-08-08T06:36:09Z")
I have an index of over 10 million documents. On those documents, I want to store an openai (or similar) embedding vector using an indexed dense vector field. I will be using cosine similarity to search through those vectors combined with filters on other fields. However, the vectors take up quite some space, storing an embedding vector for each of the more than 10 million documents will at least triple my index size. Is there a way to set a value for only part of the documents? I know that there is the "null\_value" which i've seen used for keyword fields. I don't know if its supported for dense\_vector fieldsthough, but if it is I wonder what would be a good value to be used as null value for a dense\_vector field without it affecting the search results? Documents with the null\_value shouldn't match the cosine similarity search. My ultimate goal is to be able to a normal keyword search combined on all documents like I normally do without using vector search, but in some cases do a cosine similarity search combined with filters on the other fields, which should only return documents that have an actual vector value for the dense\_vector field. Only documents added in the last 3 months will get a vector value, old documents will have their vectors removed. ---
**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\ **Post date:** [August 8, 2023, 2:09pm UTC](https://discuss.elastic.co/t/populate-a-dense-vector-field-for-only-a-subset-of-the-documents/340326/2 "2023-08-08T14:09:29Z")
Heya @sbruinsje, You can do the following to not index a vector for a document: - Not include the field value at all for the field - add the field value as `null` specifically (supported as of 8.7) In both of these causes, the vector isn't searchable and won't be considered for brute force nor approximate nearest neighbor search. Does this answer your question? ---
**Author:** ![sbruinsje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sbruinsje/32/108568_2.png) [@sbruinsje](https://discuss.elastic.co/u/sbruinsje)\ **Post date:** [August 9, 2023, 8:05am UTC](https://discuss.elastic.co/t/populate-a-dense-vector-field-for-only-a-subset-of-the-documents/340326/3 "2023-08-09T08:05:41Z")
> [@BenTrent](#): > > - Not include the field value at all for the field Thanks @BenTrent! Oh cool, there is an explicit null value supported now 🙂 So if I understand correctly, I can set the value of the dense\_vector field to null for some documents causing those documents to never be included in a knn search, yet the documents with an actual vector value for the dense\_vector field will be considered for the knn search? How does it interact with hybrid search? Lets say some documents are scored by the regular keyword search which do not have a vector value for the dense\_vector field. How is the hybrid scoring dealing with that? ---
**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\ **Post date:** [August 9, 2023, 11:45am UTC](https://discuss.elastic.co/t/populate-a-dense-vector-field-for-only-a-subset-of-the-documents/340326/4 "2023-08-09T11:45:04Z")
> [@sbruinsje](#): > > How does it interact with hybrid search? Hybrid search in Elasticsearch is currently only an "or" combination. Having documents that don't overlap between the two search kinds (BM25 & nearest vectors) is common and is perfectly fine. A document that is scored via BM25, but not in kNN, will only have its BM25 score. ---
**Author:** ![sbruinsje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sbruinsje/32/108568_2.png) [@sbruinsje](https://discuss.elastic.co/u/sbruinsje)\ **Post date:** [August 9, 2023, 12:56pm UTC](https://discuss.elastic.co/t/populate-a-dense-vector-field-for-only-a-subset-of-the-documents/340326/5 "2023-08-09T12:56:58Z")
> [@BenTrent](#): > > rlap between the two search kinds (BM25 & nearest vectors) is common and is perfectly fine. A document that is scored via BM25, but not in kNN, will only have its BM25 score. Perfect that answers my questions. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 6, 2023, 12:57pm UTC](https://discuss.elastic.co/t/populate-a-dense-vector-field-for-only-a-subset-of-the-documents/340326/6 "2023-09-06T12:57:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.