# Need Help Tuning ES for Large Data
**URL:** https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207
**Category:** Elasticsearch
**Created:** [June 28, 2017, 11:30pm UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207 "2017-06-28T23:30:08Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)
#### Post date: [June 28, 2017, 11:30pm UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/1 "2017-06-28T23:30:08Z")
Hello Everyone,
I have a relatively large data where we use a custom python script to ingest data using the bulk API. These are AWS Cloudtrail data for over 200 AWS Accounts aggregated in single s3 bucket. This has been running good since couple of years but as the data and number of AWS is getting added each day, it has become a nightmare to manage the number of shards it builds. Currently it creates a index for every account and every region. Say like bigdata-us-east- , and then us-west-2-. This goes on for every combination for all 13 AWS regions and 200 AWS accounts. So there are lots of shards getting created every single day. So if a single instance goes down it takes a really long time to rebuild it.
If someone can suggest a better way of indexing data which reduces the motherload of shards, it would really help.
My cluster health output:-
{
"cluster\_name": "elk-prod",
"status": "red",
"timed\_out": false,
"number\_of\_nodes": 7,
"number\_of\_data\_nodes": 1,
"active\_primary\_shards": 198,
"active\_shards": 198,
"relocating\_shards": 0,
"initializing\_shards": 4,
"unassigned\_shards": 29020,
"delayed\_unassigned\_shards": 0,
"number\_of\_pending\_tasks": 14,
"number\_of\_in\_flight\_fetch": 0,
"task\_max\_waiting\_in\_queue\_millis": 19148,
"active\_shards\_percent\_as\_number": 0.6775716925603997
}
FYI this is a 9 node cluster with 3 data, ingest and master node served with HA Proxy.
Please let me know if you need any further details from my cluster and i am ready to provide it.
--
Niraj
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 28, 2017, 11:31pm UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/2 "2017-06-28T23:31:55Z")
What version are you on?
---
### Author:  [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)
#### Post date: [June 28, 2017, 11:59pm UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/3 "2017-06-28T23:59:47Z")
5.2.1
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 29, 2017, 2:39am UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/4 "2017-06-29T02:39:18Z")
Use the shrink API to reduce your shard count, then look at ways to merge some of your indices.
Maybe by region, or by account but do monthly/yearly and a low shard count?
---
### Author:  [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)
#### Post date: [June 29, 2017, 5:29am UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/5 "2017-06-29T05:29:58Z")
Thanks Mark for the suggestion. Can you show me an example of shrinking. I was looking at the documentation but found it a bit complicated.
Also will the shrinking results in any data loss as these are actual audit data of customers.
--
Niraj
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 29, 2017, 10:09am UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/6 "2017-06-29T10:09:57Z")
> [@niraj\_kumar](#):
>
> Also will the shrinking results in any data loss as these are actual audit data of customers.
No it will not.
Does [And the big one said "Rollover" — Managing Elasticsearch time-based indices efficiently | Elastic Blog](https://www.elastic.co/blog/managing-time-based-indices-efficiently) help?
---
### Author:  [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)
#### Post date: [June 30, 2017, 6:45am UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/7 "2017-06-30T06:45:32Z")
Thanks for the link mark that was helpful. So i should basically create a small script and move/shrink over some old data to reduce the shard count. Right.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 30, 2017, 7:19am UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/8 "2017-06-30T07:19:59Z")
For some short term relief, yep.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 17, 2017, 9:11pm UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/9 "2017-07-17T21:11:58Z")
A quick question @niraj_kumar , are you using the default number of shards or have you customised them?
---
### Author:  [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)
#### Post date: [July 17, 2017, 9:39pm UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/10 "2017-07-17T21:39:45Z")
@warkolm I have customized them to three.
---
### Author:  [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)
#### Post date: [August 1, 2017, 4:25am UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/11 "2017-08-01T04:25:38Z")
@warkolm. Do we have a documentation for merging indices.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [August 1, 2017, 5:52am UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/12 "2017-08-01T05:52:56Z")
Indicies? No, you will need to use the reindex API for that.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 29, 2017, 5:53am UTC](https://discuss.elastic.co/t/need-help-tuning-es-for-large-data/91207/13 "2017-08-29T05:53:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.