# Handling TB's of data in Elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [September 22, 2020, 6:31am UTC](https://discuss.elastic.co/t/handling-tbs-of-data-in-elasticsearch/249455 "2020-09-22T06:31:48Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Jayabal\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jayabal_k/32/41905_2.png) [@Jayabal\_K](https://discuss.elastic.co/u/Jayabal_K)\ **Post date:** [September 22, 2020, 6:31am UTC](https://discuss.elastic.co/t/handling-tbs-of-data-in-elasticsearch/249455/1 "2020-09-22T06:31:49Z")
Hi all, In my case, I need to handle 3TB of data in elaticsearch, We have a job to rotate the index daily once. Each day a new index will be created, which is of size 100 GB. For 30 days we have 30 indices( 30 \* 100GB =3000GB). we need to create monthly reports on this using aggregation queries(mostly terms aggs). But running aggregation queries on 3 TB of data, leads to client node crash. Could anyone help me with this? Cluster details: 3 master nodes: 500 vcpu and 1GB physical memory 5 data nodes: 1vcpu and 1GB physical memory 1 client node: 500vcpu and 1GB physical memory what could be the best solution for this? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 22, 2020, 9:11am UTC](https://discuss.elastic.co/t/handling-tbs-of-data-in-elasticsearch/249455/2 "2020-09-22T09:11:00Z")
That sounds like very limited resources in terms of CPU, memory and heap for that data size, especially on the data nodes. I am not surprised you are having issues aggregating over the data set. I would recommend gradually increasing resources until the nodes fit your workload. ---
**Author:** ![Jayabal\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jayabal_k/32/41905_2.png) [@Jayabal\_K](https://discuss.elastic.co/u/Jayabal_K)\ **Post date:** [September 22, 2020, 12:15pm UTC](https://discuss.elastic.co/t/handling-tbs-of-data-in-elasticsearch/249455/3 "2020-09-22T12:15:16Z")
Thanks @Christian_Dahlqvist for your reply. what is your recommendation for this use case? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 22, 2020, 12:19pm UTC](https://discuss.elastic.co/t/handling-tbs-of-data-in-elasticsearch/249455/4 "2020-09-22T12:19:57Z")
I do not know what will be required, do would recommend gradually increasing until performance is sufficient. I would probably start with 16GB RAM and 8GB heap for the data nodes and a bit less for the coordinating only node. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 20, 2020, 12:20pm UTC](https://discuss.elastic.co/t/handling-tbs-of-data-in-elasticsearch/249455/5 "2020-10-20T12:20:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.