Help with manging storage for a large index


We are a new start-up going through development with no funding other than out our own pocket, I am hoping to seek some advice from others who are far more experienced than myself given we have only used Elasticsearch for the last month.

Currently, for development, we have 1 dedicated server handling a single ES node. That server has 128gb of ram (32gb allocated to ES) and a 2tb nvme drive.

We are indexing real-time financial data, our index count after just one week is a little over 2 Billion records. In production we understand we will need a multi-node cluster, our thoughts on this is to run 2-3 dedicated servers all of the same size and specs from the same service provider (ideally in the same rack but certainly on the same private network)

My issue is; Right now after one week of indexing, we have used 50% (1tb) of the nvme drive, its clear that we will be using atleast 4tb each month so the dedicated servers will require bigger drives to handle the multi-node cluster.

My question is;

We will be running aggregations on data roughly up-to 1 month old with most important being daily / weekly data and the older the data becomes the less queries/aggregations will be run and will become more of a search only basis (Eg: User filters data for a given day)

Indexing 4tb of data each month will become very expensive for us as a start-up if we use nvme / ssd storage because i assume each of the 3 dedicated servers we will have in the cluster will need a considerable amount of nvme drives.

For example; we have been given a quote for 3 servers each with same specs we have now but each server will come with 40tb of nvme drives in raid but that cost is approx $3k each / month!

Does anyone think we could potentially get away with using HDD or potentially a mix of nvme & hdd in the cluster so the HDD nodes are specifically for documents older than 1 month? I assume we would need to some how configure the cluster so elasticsearch automatically moves data to HDD nodes when the document is older than 1 month?

Stats after 1 week:
Index size: 2 Billion documents
disk usage: 50% of 2tb nvme
nodes: 1
hosts: 1
data ingestion: near real time
query/aggregations: near real time on daily/weekly documents

Many thanks, appreciate any help i can get :slight_smile:

The first thing I would recommend you do, if you have not already, is to optimize your mappings to minimize the index size.

For performance I would recommend that you keep all indices that are actively indexed into and the ones most heavily queried on nodes with fast nvme disks as this generally is very disk I/O intensive. Assuming your data is immutable and not updated you can use time-based indices and move older indices that are no longer written to to a different set of nodes with large amounts of slower storage. This is what is often referred to as a hot-warm architecture, but can also be extended to a hot-warm-cold architecture if needed.

This is often the most cost-effective way to handle large amounts of immutable data where the most recent data is queried most frequently.

1 Like

Thank you very much, Christian, i will look into these now and i very much appreciate your time. The methods you mention sound applicable as the data is time-based thus wont be updated once indexed.

I just tried pricing up a high-availability cluster of this size on Elastic Cloud with 2x240GB hot nodes, 2x5TB warm nodes and it works out at about $1800/mo for the whole cluster, and that's a managed service that includes support and backups and turnkey upgrades and a whole bunch of extra features built in. That seems pretty cost-effective vs the time spent doing it yourself.


@DavidTurner Thanks for the heads up on the elastic managed service, in all honesty, i believe we will migrate to the managed service once we reach production and can cover the additional overhead. We have a fair amount of paying customers on the sidelines who are waiting for the service to go live so i believe we will be looking at the managed service in the near future for sure.

Hi Christian,

I've spent the past 24 hours looking into what you suggested and for development and going into our first production run i believe that would be a very viable option and once things are up and running we will most likely migrate over to elastic managed service.

During the investigation of our index, i noticed number_of_replicas shows 1:

    "number_of_shards" : "1",
    "number_of_replicas" : "1",

Our elasticsearch.yml file on the server for development is: lab lab1 /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
http.port: 9200
discovery.type: 'single-node'
indices.query.bool.max_clause_count: 8192
search.max_buckets: 250000
action.destructive_requires_name: 'true'
reindex.remote.whitelist: '*:*'
xpack.monitoring.enabled: 'true'
xpack.monitoring.collection.enabled: 'true'
xpack.monitoring.collection.interval: 30s 'true' 'false' 'false' 'false'
xpack.watcher.enabled: 'false'
xpack.ilm.enabled: 'true'
xpack.sql.enabled: 'true'

As far as im aware i have not set any replica up so i believe the default must be 1, does this mean that we potentially have a replica of the data on the same server given we only have 1 server running elasticsearch with a single node / shard? IF that is the case then it could potentially reduce the storage if we remove the replica because our plan is to have a dedicated replica server in production that is seperate from our primary nodes.

Once again, many thanks and appreciate your time.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.