# Elasticsearch dense\_vector is taking up too much storage space！Help **URL:** \ **Category:** Elasticsearch\ **Tags:** vector-search\ **Created:** [September 4, 2024, 4:13pm UTC](https://discuss.elastic.co/t/elasticsearch-dense-vector-is-taking-up-too-much-storage-space-help/366046 "2024-09-04T16:13:48Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Andy\_Cong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_cong/32/134851_2.png) [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)\ **Post date:** [September 4, 2024, 4:13pm UTC](https://discuss.elastic.co/t/elasticsearch-dense-vector-is-taking-up-too-much-storage-space-help/366046/1 "2024-09-04T16:13:49Z")
I'm having an issue with Elasticsearch where my dense\_vector data is taking up significantly more storage space than expected. **ES Version** 8.2.3 **Data Model:** - `vector` field: Type is dense\_vector with a dimension of 1024. The data is stored as a 32-bit floating-point number, with values falling within the range of -0.1 to 0.1. Theoretically, each vector should occupy 4KB, - Indexed 1000 documents, which should result in a total storage of 4MB. **Actual Storage:** - Actual storage usage is 12MB. - The `_source` field is taking up 8MB, and I'm unsure why. **index mapping** ```json { "test_vector_v1": { "mappings": { "properties": { "vector": { "type": "dense_vector", "dims": 1024 } } } } } ``` **disk storage** ```json { "_shards": { "total": 1, "successful": 1, "failed": 0 }, "test_vector_v1": { "store_size": "12mb", "store_size_in_bytes": 12649263, "all_fields": { "total": "12mb", "total_in_bytes": 12642767, "inverted_index": { "total": "16.9kb", "total_in_bytes": 17348 }, "stored_fields": "8.1mb", "stored_fields_in_bytes": 8522858, "doc_values": "3.9mb", "doc_values_in_bytes": 4101499, "points": "1kb", "points_in_bytes": 1062, "norms": "0b", "norms_in_bytes": 0, "term_vectors": "0b", "term_vectors_in_bytes": 0 }, "fields": { "_id": { "total": "35kb", }, "_primary_term": { "total": "0b", }, "_seq_no": { "total": "2.5kb", }, "_source": { "total": "8.1mb", "total_in_bytes": 8504327, "inverted_index": { "total": "0b", "total_in_bytes": 0 }, "stored_fields": "8.1mb", "stored_fields_in_bytes": 8504327, "doc_values": "0b", "doc_values_in_bytes": 0, "points": "0b", "points_in_bytes": 0, "norms": "0b", "norms_in_bytes": 0, "term_vectors": "0b", "term_vectors_in_bytes": 0 }, "vector": { "total": "3.9mb", "total_in_bytes": 4100000, "inverted_index": { "total": "0b", "total_in_bytes": 0 }, "stored_fields": "0b", "stored_fields_in_bytes": 0, "doc_values": "3.9mb", "doc_values_in_bytes": 4100000, "points": "0b", "points_in_bytes": 0, "norms": "0b", "norms_in_bytes": 0, "term_vectors": "0b", "term_vectors_in_bytes": 0 } } } } ``` **Optimization Attempts:** - Disabled doc\_values: Not supported for dense\_vector. - Removed the `vector` field from \_source: This creates a new field, disabled \_source: but this creates a `_recovery_source` field, which also takes up 8MB. ```json PUT /test_vector_v2 { "mappings": { "properties": { "vector": { "type": "dense_vector", "dims": 1024 }, "file_id": { "type": "keyword" } }, "_source": { "includes": ["file_id"] } } } ``` **I'm looking for ways to reduce the storage overhead of my dense\_vector data in Elasticsearch. Is it feasible to achieve a 4MB storage limit? Thank you for your suggestion！！** ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 4, 2024, 5:00pm UTC](https://discuss.elastic.co/t/elasticsearch-dense-vector-is-taking-up-too-much-storage-space-help/366046/2 "2024-09-04T17:00:32Z")
Upgrade! Upgrade! Upgrade. We added a LOT of improvements since 8.2. 8.15.0 is your friend 😉 ---
**Author:** ![Andy\_Cong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_cong/32/134851_2.png) [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)\ **Post date:** [September 5, 2024, 1:28am UTC](https://discuss.elastic.co/t/elasticsearch-dense-vector-is-taking-up-too-much-storage-space-help/366046/4 "2024-09-05T01:28:04Z")
Thank your answer，besides upgrading, what other approaches can we take to optimize Elasticsearch?? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 5, 2024, 1:44am UTC](https://discuss.elastic.co/t/elasticsearch-dense-vector-is-taking-up-too-much-storage-space-help/366046/5 "2024-09-05T01:44:27Z")
Hi @Andy_Cong Question Are are you trying to figure out what the overall storage at scale? ... Are you trying to interpolate say you have a million of those? Is that what you're trying to figure out? Because at small scales elastic isn't as efficient. The way the data is laid out in segments on disk can take up more room than the actual data needs.... But at scale the efficiency gets much better... Often very small numbers of documents provide a very poor estimate of what the actual storage will be at scale... Kind of seems like what you're doing here. So why don't you put in 100,000 or 1M of them and then run Force merge on the index down to one segment and then you'll have a much better understanding what the actual disk space required is. That would be my suggestion... ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 5, 2024, 2:11am UTC](https://discuss.elastic.co/t/elasticsearch-dense-vector-is-taking-up-too-much-storage-space-help/366046/6 "2024-09-05T02:11:11Z")
Agreed but with a 8.15 version. So many changes happened in this space so it's useless to draw a conclusion on an old version IMO. Read: > **[Vector search improvements in Elasticsearch and Lucene — Search Labs](https://www.elastic.co/search-labs/blog/vector-search-improvements)** > > Looking back at Elastic's vector search innovations in Elasticsearch and Lucene ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 5, 2024, 2:52am UTC](https://discuss.elastic.co/t/elasticsearch-dense-vector-is-taking-up-too-much-storage-space-help/366046/7 "2024-09-05T02:52:25Z")
@dadoonet Totally agree!!!.. Plus Quantization etc etc... Then I would still load 1M and Force merge to get true size ---
**Author:** ![Andy\_Cong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_cong/32/134851_2.png) [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)\ **Post date:** [September 12, 2024, 12:51pm UTC](https://discuss.elastic.co/t/elasticsearch-dense-vector-is-taking-up-too-much-storage-space-help/366046/8 "2024-09-12T12:51:50Z")
Thank you very much. Is there any stress test performance comparison between these versions? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 12, 2024, 1:58pm UTC](https://discuss.elastic.co/t/elasticsearch-dense-vector-is-taking-up-too-much-storage-space-help/366046/9 "2024-09-12T13:58:35Z")
Nightly benchmarks [https://elasticsearch-benchmarks.elastic.co/#tracks/dense\_vector/nightly/default/90d](https://elasticsearch-benchmarks.elastic.co/#tracks/dense_vector/nightly/default/90d) ---
**Author:** ![Andy\_Cong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_cong/32/134851_2.png) [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)\ **Post date:** [September 24, 2024, 7:39am UTC](https://discuss.elastic.co/t/elasticsearch-dense-vector-is-taking-up-too-much-storage-space-help/366046/10 "2024-09-24T07:39:06Z")
Thank you. Very nice. Does this table show the ES version? How can I check the corresponding relationship? ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cb28d1e5aea7c5eba7196195ba9d7965b1a5abab.png)