# Understanding Storage Overhead in Elasticsearch for Vector Data
**URL:** https://discuss.elastic.co/t/understanding-storage-overhead-in-elasticsearch-for-vector-data/364319
**Category:** Elasticsearch
**Tags:** vector-search
**Created:** [August 4, 2024, 4:37am UTC](https://discuss.elastic.co/t/understanding-storage-overhead-in-elasticsearch-for-vector-data/364319 "2024-08-04T04:37:59Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)
#### Post date: [August 4, 2024, 4:37am UTC](https://discuss.elastic.co/t/understanding-storage-overhead-in-elasticsearch-for-vector-data/364319/1 "2024-08-04T04:37:59Z")
**Problem Description:**
This is the Elasticsearch index mapping. Theoretically, storing a vector with `float32` values should take up 4KB of storage. Therefore, each document record should occupy approximately 4KB.
However, after writing 7.5 million records, the total storage used is 90GB, with 60GB for data and 30GB for `doc_values` , which is about 1/3 of the storage. Each record occupies 12.58KB.
- chunk mapping
```json
PUT /chunk_vector
{
"mappings": {
"dynamic": "strict",
"properties": {
"chunk_id": {
"type": "long"
},
"file_id": {
"type": "keyword"
},
"file_name": {
"type": "keyword",
"doc_values": false
},
"group_id": {
"type": "keyword"
},
"vector": {
"type": "dense_vector",
"dims": 1024
},
"ctime": {
"type": "long"
},
"mtime": {
"type": "long"
}
}
}
}
```
- chunk storage:
```auto
"chunk_vector": {
"store_size": "89.9gb",
"store_size_in_bytes": 96627326548,
"all_fields": {
"total": "89.9gb",
"total_in_bytes": 96616074283,
"inverted_index": {
"total": "53.5mb",
"total_in_bytes": 56200185
},
"stored_fields": "60.1gb",
"stored_fields_in_bytes": 64585932961,
"doc_values": "29.7gb",
"doc_values_in_bytes": 31897294917,
"points": "73mb",
"points_in_bytes": 76646220,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
```
**I would like to optimize the storage of vectors. Please provide some suggestions and help resolve my queries. Thank you.**
Questions:
1. For vector search, is it necessary to use the `doc_values` feature?
2. Why does each record occupy 12.58KB?
3. How can we optimize vector storage space while keeping the vector dimension at 1024?
note: elasticsearch versoin: 8.2.3
---
### Author:  [@xeraa](https://discuss.elastic.co/u/xeraa)
#### Post date: [August 4, 2024, 5:25am UTC](https://discuss.elastic.co/t/understanding-storage-overhead-in-elasticsearch-for-vector-data/364319/2 "2024-08-04T05:25:25Z")
1. That's a really old version of Elasticsearch. Especially around dense\_vector there have been a lot of improvements so I'd strongly recommend to upgrade to a more recent version.
2. You'll have the dense\_vector both in the indexed datastructure (HNSW) and the \_source. So you could either exclude it from \_source (but then you won't be able to reindex the data any more as one major downside of that approach). Or you could enable synthetic source (but that is not a GA feature for non-TSDB indices). See [Tune approximate kNN search | Elasticsearch Guide [8.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-knn-search.html#_exclude_vector_fields_from_source) for recommendations and tradeoffs.
3. You have more fields, so each document will be larger than just the dense\_vector. But `_disk_usage` should give you a pretty good rundown of where the space is used. My guess would be that synthetic source or excluding the dense\_vector from source would be the biggest improvement you can make here; every other tuning will only give you some smaller improvements and also tradeoffs (like compression, tuning field mappings,...)
---
### Author:  [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)
#### Post date: [August 4, 2024, 10:17am UTC](https://discuss.elastic.co/t/understanding-storage-overhead-in-elasticsearch-for-vector-data/364319/3 "2024-08-04T10:17:58Z")
First,thank you for your answer , I still have a question.
1. I'll try it later.
2. good idea, I'll understand it better later.
3. I used the `_disk_usage` query and found that the `_source` field occupies 60GB, including `doc_values` . Additionally, why does the `vector` field separately occupy another 30GB?
```json
{
"_shards": {
"total": 90,
"successful": 90,
"failed": 0
},
"chunk_vector": {
"store_size": "91.4gb",
"store_size_in_bytes": 98165022949,
"all_fields": {
"total": "91.4gb",
"total_in_bytes": 98153614161,
"inverted_index": {
"total": "54.7mb",
"total_in_bytes": 57380635
},
"stored_fields": "61.1gb",
"stored_fields_in_bytes": 65610336099,
"doc_values": "30.1gb",
"doc_values_in_bytes": 32407647278,
"points": "74.6mb",
"points_in_bytes": 78250149,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"fields": {
"__soft_deletes": {
"total": "282.8kb",
"total_in_bytes": 289676,
"inverted_index": {
"total": "0b",
"total_in_bytes": 0
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "282.8kb",
"doc_values_in_bytes": 289676,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"_field_names": {
"total": "1mb",
"total_in_bytes": 1050826,
"inverted_index": {
"total": "1mb",
"total_in_bytes": 1050826
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "0b",
"doc_values_in_bytes": 0,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"_id": {
"total": "334.6mb",
"total_in_bytes": 350866540,
"inverted_index": {
"total": "39.4mb",
"total_in_bytes": 41361917
},
"stored_fields": "295.1mb",
"stored_fields_in_bytes": 309504623,
"doc_values": "0b",
"doc_values_in_bytes": 0,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"_primary_term": {
"total": "0b",
"total_in_bytes": 0,
"inverted_index": {
"total": "0b",
"total_in_bytes": 0
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "0b",
"doc_values_in_bytes": 0,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"_routing": {
"total": "174mb",
"total_in_bytes": 182494447,
"inverted_index": {
"total": "1.6mb",
"total_in_bytes": 1775801
},
"stored_fields": "172.3mb",
"stored_fields_in_bytes": 180718646,
"doc_values": "0b",
"doc_values_in_bytes": 0,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"_seq_no": {
"total": "34.4mb",
"total_in_bytes": 36087956,
"inverted_index": {
"total": "0b",
"total_in_bytes": 0
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "15.7mb",
"doc_values_in_bytes": 16466033,
"points": "18.7mb",
"points_in_bytes": 19621923,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"_source": {
"total": "60.6gb",
"total_in_bytes": 65120112830,
"inverted_index": {
"total": "0b",
"total_in_bytes": 0
},
"stored_fields": "60.6gb",
"stored_fields_in_bytes": 65120112830,
"doc_values": "0b",
"doc_values_in_bytes": 0,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"_tombstone": {
"total": "1kb",
"total_in_bytes": 1046,
"inverted_index": {
"total": "0b",
"total_in_bytes": 0
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "1kb",
"doc_values_in_bytes": 1046,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"_version": {
"total": "96kb",
"total_in_bytes": 98353,
"inverted_index": {
"total": "0b",
"total_in_bytes": 0
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "96kb",
"doc_values_in_bytes": 98353,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"chunk_id": {
"total": "38.6mb",
"total_in_bytes": 40555844,
"inverted_index": {
"total": "0b",
"total_in_bytes": 0
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "12.9mb",
"doc_values_in_bytes": 13570698,
"points": "25.7mb",
"points_in_bytes": 26985146,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"ctime": {
"total": "26.3mb",
"total_in_bytes": 27680959,
"inverted_index": {
"total": "0b",
"total_in_bytes": 0
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "11.3mb",
"doc_values_in_bytes": 11859476,
"points": "15mb",
"points_in_bytes": 15821483,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"file_id": {
"total": "13mb",
"total_in_bytes": 13662392,
"inverted_index": {
"total": "4.3mb",
"total_in_bytes": 4588082
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "8.6mb",
"doc_values_in_bytes": 9074310,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"file_name": {
"total": "6.5mb",
"total_in_bytes": 6827886,
"inverted_index": {
"total": "6.5mb",
"total_in_bytes": 6827886
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "0b",
"doc_values_in_bytes": 0,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"group_id": {
"total": "6.8mb",
"total_in_bytes": 7155203,
"inverted_index": {
"total": "1.6mb",
"total_in_bytes": 1776123
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "5.1mb",
"doc_values_in_bytes": 5379080,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"mtime": {
"total": "26.3mb",
"total_in_bytes": 27681073,
"inverted_index": {
"total": "0b",
"total_in_bytes": 0
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "11.3mb",
"doc_values_in_bytes": 11859476,
"points": "15mb",
"points_in_bytes": 15821597,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
},
"vector": {
"total": "30.1gb",
"total_in_bytes": 32339049130,
"inverted_index": {
"total": "0b",
"total_in_bytes": 0
},
"stored_fields": "0b",
"stored_fields_in_bytes": 0,
"doc_values": "30.1gb",
"doc_values_in_bytes": 32339049130,
"points": "0b",
"points_in_bytes": 0,
"norms": "0b",
"norms_in_bytes": 0,
"term_vectors": "0b",
"term_vectors_in_bytes": 0
}
}
}
}
```
---
### Author:  [@xeraa](https://discuss.elastic.co/u/xeraa)
#### Post date: [August 5, 2024, 7:52am UTC](https://discuss.elastic.co/t/understanding-storage-overhead-in-elasticsearch-for-vector-data/364319/4 "2024-08-05T07:52:12Z")
For point 3: You have the full document in \_source (by default). And on top you need to store fields in the indexed data structure. That's why you could either exclude the large dense\_vector field from source or use synthetic source — though both have their tradeoffs.