# Tips on Optimization **URL:** \ **Category:** Elasticsearch\ **Created:** [October 9, 2017, 9:20am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199 "2017-10-09T09:20:11Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![aarjan](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@aarjan](https://discuss.elastic.co/u/aarjan)\ **Post date:** [October 9, 2017, 9:20am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199/1 "2017-10-09T09:20:11Z")
I have just uploaded 4 years of 4GB data into aws elastic cluster. I have indexed documents by day. Since, i had stuck with default shard size of 5, now it has grown into more than 7300 primary shards. I had also stick with the dynamic mapping of es, but i wanted to make my own template now. I have thought of different options to decrease the shard size, 1. Reindexing data one index at a time - Need to write a script to reindex each one 2.Shrinking index - I don't think i can change the mapping of the index. - Still, i have to index one by one. 1. Bulk indexing. - I don't think it would be good idea to add a index settings/mapping line to each document record. (There are roughly 2 million documents) 1. Uploading the data again 4.a Using Logstash with es in input and output 4.b Uploading from scratch ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 9, 2017, 9:33am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199/2 "2017-10-09T09:33:36Z")
That is indeed [far too many shards](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster). If you only have 4GB of data, you should be fine using a yearly index with 1 primary shard. It may actually be easier and more efficient to delete it all and index it again from scratch using the correct template rather than trying to reindex that many indices. ---
**Author:** ![aarjan](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@aarjan](https://discuss.elastic.co/u/aarjan)\ **Post date:** [October 9, 2017, 9:39am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199/3 "2017-10-09T09:39:27Z")
Will that be hard for searching or managing data... if we indexed yearly? Btw, kibana discover has pretty good visualizations with timeperiod. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 9, 2017, 9:41am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199/4 "2017-10-09T09:41:13Z")
You might get away with monthly indices as well, but it all depends on your cluster spec and data volume. Having lots of very small shards is very inefficient. ---
**Author:** ![aarjan](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@aarjan](https://discuss.elastic.co/u/aarjan)\ **Post date:** [October 9, 2017, 9:47am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199/5 "2017-10-09T09:47:23Z")
Could we index the data quarterly, since, we are pushing it through logstash. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 9, 2017, 9:48am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199/6 "2017-10-09T09:48:55Z")
What is your total data volume per year? How long do you plan to keep the data? ---
**Author:** ![aarjan](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@aarjan](https://discuss.elastic.co/u/aarjan)\ **Post date:** [October 9, 2017, 9:52am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199/7 "2017-10-09T09:52:56Z")
It is about 2-3 gb/year. We are a growing company, and our growth is more than 20%/year. We plan to keep the data atleast for 5-10years. It depends on team demand, but they will expect to cover the data of more than 5 years, mostly for visualizations. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 9, 2017, 9:57am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199/8 "2017-10-09T09:57:45Z")
I would recommend going for a yearly index with 1 or 2 primary shard initially. If you realise that shards are getting too big later on you can easily adjust it then and switch to a higher number of primary shards or monthly/quarterly indices. ---
**Author:** ![aarjan](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@aarjan](https://discuss.elastic.co/u/aarjan)\ **Post date:** [October 9, 2017, 10:02am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199/9 "2017-10-09T10:02:10Z")
Thank you very much for you advice. And I don't think, it would be a problem to query the index using wildcard.... if we plan to change the index pattern to month/quarterly later on Sorry to poke you again, how can i write the index pattern to correspond to quaterly index pattern, in Logstash es ouput.... ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 9, 2017, 10:04am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199/10 "2017-10-09T10:04:44Z")
As long as you have the same prefix that will not be a problem. Am however not sure how to create a quarterly pattern in Logstash. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 6, 2017, 10:05am UTC](https://discuss.elastic.co/t/tips-on-optimization/103199/11 "2017-11-06T10:05:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.