# Capacity Planning Guidelines? (estimating index size) **URL:** \ **Category:** Elasticsearch\ **Created:** [October 20, 2011, 2:57pm UTC](https://discuss.elastic.co/t/capacity-planning-guidelines-estimating-index-size/5650 "2011-10-20T14:57:49Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Schnyder](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Schnyder](https://discuss.elastic.co/u/Schnyder)\ **Post date:** [October 20, 2011, 2:57pm UTC](https://discuss.elastic.co/t/capacity-planning-guidelines-estimating-index-size/5650/1 "2011-10-20T14:57:49Z")
We're kicking off a project that will involve indexing terrabytes of data. We're considering using ElasticSearch for the job. However I need to determine the hardware requirements to hold such a large index. Are there any guidelines to help estimate the size of an index relative to the size of the source data? For instance, if index 100MB of new JSON data, how much can I expect ElasticSearch's index to grow as a result? Any advice would be GREATLY appreciated. Thanks, Chris ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [October 20, 2011, 8:27pm UTC](https://discuss.elastic.co/t/capacity-planning-guidelines-estimating-index-size/5650/2 "2011-10-20T20:27:46Z")
The hardware requirements also depend on what you want to do with it e.g. how much traffic? As a rule of thumb I would say that a lucene index is a bit smaller than the actual data. BUT it really depends on what things of the data should be indexed or if there are stored field, if you use the \_all field or the \_sources etc. I would suggest to setup a test index of those 100MB and see it in real life. Also: if you are about to index the things into one index it will get slower and slower, so maybe you setup some index rolling mechanism (or play with the shard count) - especially if this is none-static data. On 20 Okt., 16:57, Schnyder [chris.schny...@cardinal-holdings.com](mailto:chris.schny...@cardinal-holdings.com) wrote: > We're kicking off a project that will involve indexing terrabytes of data. > We're considering using Elasticsearch for the job. However I need to > determine the hardware requirements to hold such a large index. > > Are there any guidelines to help estimate the size of an index relative to > the size of the source data? For instance, if index 100MB of new JSON data, > how much can I expect Elasticsearch's index to grow as a result? > > Any advice would be GREATLY appreciated. > > Thanks, > Chris ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [October 20, 2011, 11:28pm UTC](https://discuss.elastic.co/t/capacity-planning-guidelines-estimating-index-size/5650/3 "2011-10-20T23:28:30Z")
Hi, While this won't answer all your questions directly (there is no exact answer without knowing all details and, really, without doing some tests), have a look at the disk & memory size estimator for Lucene/ Solr - [http://search-lucene.com/?q=size+estimator&fc\_project=Lucene&fc\_project=Solr](http://search-lucene.com/?q=size+estimator&fc_project=Lucene&fc_project=Solr) . Parts of this will be applicable to Elasticsearch, but of course even this estimator is not perfect. ## Otis Check out Search Analytics SaaS - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) On Oct 20, 10:57 am, Schnyder [chris.schny...@cardinal-holdings.com](mailto:chris.schny...@cardinal-holdings.com) wrote: > We're kicking off a project that will involve indexing terrabytes of data. > We're considering using Elasticsearch for the job. However I need to > determine the hardware requirements to hold such a large index. > > Are there any guidelines to help estimate the size of an index relative to > the size of the source data? For instance, if index 100MB of new JSON data, > how much can I expect Elasticsearch's index to grow as a result? > > Any advice would be GREATLY appreciated. > > Thanks, > Chris > > -- > View this message in context:[http://elasticsearch-users.115913.n3.nabble.com/Capacity-Planning-Gui](http://elasticsearch-users.115913.n3.nabble.com/Capacity-Planning-Gui)... > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 21, 2011, 1:17am UTC](https://discuss.elastic.co/t/capacity-planning-guidelines-estimating-index-size/5650/4 "2011-10-21T01:17:34Z")
You will have to do some capacity tests with a smaller set of data. Some points to think about: 1. By default, \_source is stored (the actual json you added). It usually make sense to turn on compression for it. 2. \_all is by default enabled, meaning that on top of all the specific fields being indexed, another field which aggregates all of them is also indexed. It makes searching much simpler, but does add an overhead. You can completely disable it or pick and choose in the mappings if fields should be included in all or not. 3. You might not need to index all the json fields, if you have some that you don't need to index, you can map those with index set to no. -shay.banon On Thu, Oct 20, 2011 at 4:57 PM, Schnyder \< [chris.schnyder@cardinal-holdings.com](mailto:chris.schnyder@cardinal-holdings.com)\> wrote: > We're kicking off a project that will involve indexing terrabytes of data. > We're considering using Elasticsearch for the job. However I need to > determine the hardware requirements to hold such a large index. > > Are there any guidelines to help estimate the size of an index relative to > the size of the source data? For instance, if index 100MB of new JSON > data, > how much can I expect Elasticsearch's index to grow as a result? > > Any advice would be GREATLY appreciated. > > Thanks, > Chris > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Capacity-Planning-Guidelines-estimating-index-size-tp3437936p3437936.html](http://elasticsearch-users.115913.n3.nabble.com/Capacity-Planning-Guidelines-estimating-index-size-tp3437936p3437936.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:51am UTC](https://discuss.elastic.co/t/capacity-planning-guidelines-estimating-index-size/5650/5 "2017-07-06T03:51:16Z")