# Performance issue on 40TB index
**URL:** https://discuss.elastic.co/t/performance-issue-on-40tb-index/376021
**Category:** Elasticsearch
**Created:** [March 17, 2025, 3:13pm UTC](https://discuss.elastic.co/t/performance-issue-on-40tb-index/376021 "2025-03-17T15:13:58Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Suresh\_Ghatuwa](https://discuss.elastic.co/u/Suresh_Ghatuwa)
#### Post date: [March 17, 2025, 3:13pm UTC](https://discuss.elastic.co/t/performance-issue-on-40tb-index/376021/1 "2025-03-17T15:13:58Z")
Hello All,
I have 40 TB of index having about 6 Billion documents in single index.
My ES query is
Fetching 100000 unique values of **uniqueId** values (applying terms aggregations) from single ES query.
Currently, we initialized index having 900 shards across 35 data nodes. And most of the time is spending on coordinating nodes.
How can i configure the Elasticsearch cluster for better performance?
Please suggest.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 17, 2025, 3:49pm UTC](https://discuss.elastic.co/t/performance-issue-on-40tb-index/376021/2 "2025-03-17T15:49:59Z")
What type of data do you have in that index? What is the structure of the data? What does the query/aggregation look like? How many unique ids are there in total?
Which version of Elasticsearch are you using?
> [@Suresh\_Ghatuwa](#):
>
> And most of the time is spending on coordinating nodes.
How have you determined this? What is the specification of the nodes? What exactly is the performance issue? What latency are you experiencing?
> [@Suresh\_Ghatuwa](#):
>
> I have 40 TB of index having about 6 Billion documents in single index.
Is this the size of primary and replica shards? If so, how many replica shards do you have configured?
---
### Author:  [@Suresh\_Ghatuwa](https://discuss.elastic.co/u/Suresh_Ghatuwa)
#### Post date: [March 25, 2025, 4:24am UTC](https://discuss.elastic.co/t/performance-issue-on-40tb-index/376021/3 "2025-03-25T04:24:16Z")
@Christian_Dahlqvist
Please find the reply:
What type of data do you have in that index? What is the structure of the data?
> > Currently, we are storing all data (Total 6.3 Billion) in single index. We are storing as flat data with around 500 fields in each document. And we are storing the different type of data in single index.
What does the query/aggregation look like?
> > We are aggregating the total amount of each unique ids.
How many unique ids are there in total?
> > There present around 11M unique ids in single type.
Which version of Elasticsearch are you using?
> > We are using ES 7.17
How have you determined this?
> > We executed the ES profiling API and found the response of shards is less than 1 sec. And usage of data node is low while the usage of coordinating node is high.
What is the specification of the nodes?
> > We are using c7g.8xlarge (64 GB total memory and 32 vCPU) for data node and coordinating node.
What exactly is the performance issue? What latency are you experiencing?
> > On analyzing the profiling response, the time taking section is from Coordinating node and the response time from ES is high.
Is this the size of primary and replica shards? If so, how many replica shards do you have configured?
> > Currently we are using primary shards only (i.e. without replica shards). Does replica shards also improved on performance?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 25, 2025, 6:40am UTC](https://discuss.elastic.co/t/performance-issue-on-40tb-index/376021/4 "2025-03-25T06:40:38Z")
> [@Suresh\_Ghatuwa](#):
>
> We are using c7g.8xlarge (64 GB total memory and 32 vCPU) for data node and coordinating node.
What does CPU usage look like on the different node types when you run a query? What size and type of storage do you have attached?
Elasticsearch is generally limited by disk I/O and not CPU, so I tend to use memory optimised instances for Elasticsearch clusters unless I am running a lot of CPU heavy processing, e.g. complex ingest pipelines. Have you run `iostat -x` on the data nodes when you are querying to verify that the storage is not a limiting factor (the coordinating node can only process data as fast as it comes off the data nodes after all)?
---
### Author:  [@Suresh\_Ghatuwa](https://discuss.elastic.co/u/Suresh_Ghatuwa)
#### Post date: [March 25, 2025, 7:43am UTC](https://discuss.elastic.co/t/performance-issue-on-40tb-index/376021/5 "2025-03-25T07:43:11Z")
Regarding the CPU usage, usage on data node is normal. But high CPU and Memory in Coordinating nodes.
Currently, we are using st1 disk type.
Please find the result of one of the data node from `iostat -x`
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 25, 2025, 8:52am UTC](https://discuss.elastic.co/t/performance-issue-on-40tb-index/376021/6 "2025-03-25T08:52:33Z")
What is high and normal in terms of concrete numbers? How many cores are fully utilised?