# Vector search for large amount of data with limited RAM resources
**URL:** https://discuss.elastic.co/t/vector-search-for-large-amount-of-data-with-limited-ram-resources/337681
**Category:** Elasticsearch
**Tags:** docker
**Created:** [July 5, 2023, 2:04pm UTC](https://discuss.elastic.co/t/vector-search-for-large-amount-of-data-with-limited-ram-resources/337681 "2023-07-05T14:04:39Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@louis\_sg](https://discuss.elastic.co/u/louis_sg)
#### Post date: [July 5, 2023, 2:04pm UTC](https://discuss.elastic.co/t/vector-search-for-large-amount-of-data-with-limited-ram-resources/337681/1 "2023-07-05T14:04:39Z")
Hi there,
I am new here trying to use Elasticsearch for vector similarity search.
My dataset is as large as 100M records, each containing a 384 dimensions vector and a string payload.
I am building the HNSW index type, but it will raise OOM errors even when I use a small portion (10M) of my data in a 25Gb docker container. Considering my machine's total RAM available, it might be hard to increase RAM to fit all 100M records.
Any suggestion on uploading all data into the database with limited RAM resources?
---
### Author:  [@Alex\_Salgado-Elastic](https://discuss.elastic.co/u/Alex_Salgado-Elastic)
#### Post date: [July 5, 2023, 2:24pm UTC](https://discuss.elastic.co/t/vector-search-for-large-amount-of-data-with-limited-ram-resources/337681/2 "2023-07-05T14:24:23Z")
> [@louis\_sg](#):
>
> Hi there,
> I am new here trying to use Elasticsearch for vector similarity search.
> My dataset is as large as 100M records, each containing a 384 dimensions vector and a string payload.
> I am building the HNSW index type, but it will raise OOM errors even when I use a small portion (10M) of my data in a 25Gb docker container. Considering my machine's total RAM available, it might be hard to increase RAM to fit all 100M records.
> Any suggestion on uploading all data into the database with limited RAM resources?
Hi @louis_sg , have you tried this solution from stackoverflow ?[indexing - Uploading large 800gb json file from remote server to elasticsearch - Stack Overflow](https://stackoverflow.com/questions/32465870/uploading-large-800gb-json-file-from-remote-server-to-elasticsearch)
Split your data into smaller chunks and send them to Elasticsearch using multiple Bulk Requests.
---
### Author:  [@louis\_sg](https://discuss.elastic.co/u/louis_sg)
#### Post date: [July 5, 2023, 3:10pm UTC](https://discuss.elastic.co/t/vector-search-for-large-amount-of-data-with-limited-ram-resources/337681/3 "2023-07-05T15:10:57Z")
Yes, I am already using this method to upload records in batches of 64.
---
### Author:  [@Alex\_Salgado-Elastic](https://discuss.elastic.co/u/Alex_Salgado-Elastic)
#### Post date: [July 5, 2023, 4:03pm UTC](https://discuss.elastic.co/t/vector-search-for-large-amount-of-data-with-limited-ram-resources/337681/4 "2023-07-05T16:03:15Z")
Is it possible to show your mappings about the field that you are using to create the embeddings? and the max size of it?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 2, 2023, 4:04pm UTC](https://discuss.elastic.co/t/vector-search-for-large-amount-of-data-with-limited-ram-resources/337681/5 "2023-08-02T16:04:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.