# Help with manging storage for a large index **URL:** \ **Category:** Elasticsearch\ **Tags:** datastreams\ **Created:** [May 7, 2021, 9:00pm UTC](https://discuss.elastic.co/t/help-with-manging-storage-for-a-large-index/272447 "2021-05-07T21:00:44Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![predev](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@predev](https://discuss.elastic.co/u/predev)\ **Post date:** [May 7, 2021, 9:00pm UTC](https://discuss.elastic.co/t/help-with-manging-storage-for-a-large-index/272447/1 "2021-05-07T21:00:44Z")
Hi, We are a new start-up going through development with no funding other than out our own pocket, I am hoping to seek some advice from others who are far more experienced than myself given we have only used Elasticsearch for the last month. Currently, for development, we have 1 dedicated server handling a single ES node. That server has 128gb of ram (32gb allocated to ES) and a 2tb nvme drive. We are indexing real-time financial data, our index count after just one week is a little over 2 Billion records. In production we understand we will need a multi-node cluster, our thoughts on this is to run 2-3 dedicated servers all of the same size and specs from the same service provider (ideally in the same rack but certainly on the same private network) My issue is; Right now after one week of indexing, we have used 50% (1tb) of the nvme drive, its clear that we will be using atleast 4tb each month so the dedicated servers will require bigger drives to handle the multi-node cluster. My question is; We will be running aggregations on data roughly up-to 1 month old with most important being daily / weekly data and the older the data becomes the less queries/aggregations will be run and will become more of a search only basis (Eg: User filters data for a given day) Indexing 4tb of data each month will become very expensive for us as a start-up if we use nvme / ssd storage because i assume each of the 3 dedicated servers we will have in the cluster will need a considerable amount of nvme drives. For example; we have been given a quote for 3 servers each with same specs we have now but each server will come with 40tb of nvme drives in raid but that cost is approx $3k each / month! Does anyone think we could potentially get away with using HDD or potentially a mix of nvme & hdd in the cluster so the HDD nodes are specifically for documents older than 1 month? I assume we would need to some how configure the cluster so elasticsearch automatically moves data to HDD nodes when the document is older than 1 month? Stats after 1 week: Index size: 2 Billion documents disk usage: 50% of 2tb nvme nodes: 1 hosts: 1 data ingestion: near real time query/aggregations: near real time on daily/weekly documents Many thanks, appreciate any help i can get 🙂 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 7, 2021, 10:04pm UTC](https://discuss.elastic.co/t/help-with-manging-storage-for-a-large-index/272447/2 "2021-05-07T22:04:23Z")
The first thing I would recommend you do, if you have not already, is to [optimize your mappings to minimize the index size](https://www.elastic.co/guide/en/elasticsearch/reference/7.12/tune-for-disk-usage.html). For performance I would recommend that you keep all indices that are actively indexed into and the ones most heavily queried on nodes with fast nvme disks as this generally is very disk I/O intensive. Assuming your data is immutable and not updated you can use time-based indices and move older indices that are no longer written to to a different set of nodes with large amounts of slower storage. This is what is often referred to as [a hot-warm architecture](https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management), but can also be extended to a hot-warm-cold architecture if needed. This is often the most cost-effective way to handle large amounts of immutable data where the most recent data is queried most frequently. ---
**Author:** ![predev](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@predev](https://discuss.elastic.co/u/predev)\ **Post date:** [May 7, 2021, 10:17pm UTC](https://discuss.elastic.co/t/help-with-manging-storage-for-a-large-index/272447/3 "2021-05-07T22:17:13Z")
Thank you very much, Christian, i will look into these now and i very much appreciate your time. The methods you mention sound applicable as the data is time-based thus wont be updated once indexed. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 8, 2021, 8:19am UTC](https://discuss.elastic.co/t/help-with-manging-storage-for-a-large-index/272447/4 "2021-05-08T08:19:05Z")
> [@predev](#): > > For example; we have been given a quote for 3 servers each with same specs we have now but each server will come with 40tb of nvme drives in raid but that cost is approx $3k each / month! I just tried pricing up a high-availability cluster of this size on [Elastic Cloud](https://cloud.elastic.co/pricing) with 2x240GB hot nodes, 2x5TB warm nodes and it works out at about $1800/mo for the whole cluster, and that's a managed service that includes support and backups and turnkey upgrades and a whole bunch of extra features built in. That seems pretty cost-effective vs the time spent doing it yourself. ---
**Author:** ![predev](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@predev](https://discuss.elastic.co/u/predev)\ **Post date:** [May 8, 2021, 5:19pm UTC](https://discuss.elastic.co/t/help-with-manging-storage-for-a-large-index/272447/5 "2021-05-08T17:19:39Z")
@DavidTurner Thanks for the heads up on the elastic managed service, in all honesty, i believe we will migrate to the managed service once we reach production and can cover the additional overhead. We have a fair amount of paying customers on the sidelines who are waiting for the service to go live so i believe we will be looking at the managed service in the near future for sure. ---
**Author:** ![predev](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@predev](https://discuss.elastic.co/u/predev)\ **Post date:** [May 8, 2021, 5:32pm UTC](https://discuss.elastic.co/t/help-with-manging-storage-for-a-large-index/272447/6 "2021-05-08T17:32:14Z")
Hi Christian, I've spent the past 24 hours looking into what you suggested and for development and going into our first production run i believe that would be a very viable option and once things are up and running we will most likely migrate over to elastic managed service. During the investigation of our index, i noticed number\_of\_replicas shows 1: ``` "number_of_shards" : "1", "number_of_replicas" : "1", ``` Our elasticsearch.yml file on the server for development is: ``` cluster.name: lab node.name: lab1 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory_lock: true network.host: 0.0.0.0 http.port: 9200 discovery.type: 'single-node' indices.query.bool.max_clause_count: 8192 search.max_buckets: 250000 action.destructive_requires_name: 'true' reindex.remote.whitelist: '*:*' xpack.monitoring.enabled: 'true' xpack.monitoring.collection.enabled: 'true' xpack.monitoring.collection.interval: 30s xpack.security.enabled: 'true' xpack.security.audit.enabled: 'false' node.ml: 'false' xpack.ml.enabled: 'false' xpack.watcher.enabled: 'false' xpack.ilm.enabled: 'true' xpack.sql.enabled: 'true' ``` As far as im aware i have not set any replica up so i believe the default must be 1, does this mean that we potentially have a replica of the data on the same server given we only have 1 server running elasticsearch with a single node / shard? IF that is the case then it could potentially reduce the storage if we remove the replica because our plan is to have a dedicated replica server in production that is seperate from our primary nodes. Once again, many thanks and appreciate your time. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 5, 2021, 5:32pm UTC](https://discuss.elastic.co/t/help-with-manging-storage-for-a-large-index/272447/7 "2021-06-05T17:32:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.