# Size of Index

**URL:** <https://discuss.elastic.co/t/size-of-index/43065>\
**Category:** Elasticsearch\
**Created:** [March 1, 2016, 2:45am UTC](https://discuss.elastic.co/t/size-of-index/43065 "2016-03-01T02:45:36Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\
**Post date:** [March 1, 2016, 2:45am UTC](https://discuss.elastic.co/t/size-of-index/43065/1 "2016-03-01T02:45:36Z")

</div>

I'm trying a simple test to understand the size of index base on what I observed.

Case 1: Total Indexed Volume 15 Million Documents of size (74GB) . Index size 38.1 GB.  
Case 2: Total Indexed Volume 500K Documents of size (3 GB) . Index size 18 GB.

Case 1 is Great compression where as Case 2 is opposite way. The 500K is a subset for 15 Millon. If my understanding is correct it is because of repetitive terms that come from analyzed field.

Question 1: Can someone clarify please?

Also, on other note, I used a single document and created 3 versions of index (0 replica, 1 shard) based on same document, which is size 4 KB in raw.  
v1 - Analyzed on single attribute  
v2 - Analyzed on single attribute, but \_all is set to fals  
v3 - No attribute is analyzed

When I put the content, below is what the output I saw

# get \_cat/shards/v1,v2,v3?v

index shard prirep state docs store ip node  
v1 0 p STARTED 5 18.8kb 127.0.0.1 Wildboys  
v2 0 p STARTED 5 18.8kb 127.0.0.1 Wildboys  
v2 0 p STARTED 5 19kb 127.0.0.1 Wildboys

It would be helpful if someone clarifies below queries

Question 2 : How is it that size is so greater than original text. I tried doing /v1/\_analyze...on analyzed content and it translates to 18 terms

Question 3: Why docs is 5. get \_cat/indices/v1,v2,v3?v also says 5 as document count, though it is only one. get /v1/\_count says correctly as 1.

Question 4: What is a recommended size of Shard and how many shard we could have? \<=50 GB on a 14GB RAM machine? Is there any logic for computing the same.

Question 5: Any specific options to reduce size of index other than below  
\_source=false, which I cannot as I'm not storing fields individually and would avoid  
\_all=False. I have only one analyzed field. Rest all is not\_analyzed

NOTE: I referred below URLs for validating various items

> **[Sizing Elasticsearch
	  	 | Elastic](https://www.elastic.co/blog/found-sizing-elasticsearch)**
>
> We're often asked 'How big a cluster do I need?', and it's usually hard to be more specific than 'Well, it depends!'. There are so many variables, where knowledge about your application's specific workload and your performance expectations are just...

  

> **[Elasticsearch Indexing Performance Cheatsheet - codecentric AG Blog](https://blog.codecentric.de/en/2014/05/elasticsearch-indexing-performance-cheatsheet/)**
>
> You plan to index large amounts of data in Elasticsearch? Or you are already trying to do so but it turns out that throughput is too low? Here is a collection of tips and ideas to increase indexing throughput with Elasticsearch. Some of them I have...

My goal is to get to 20 Million documents/day and keep it for at-least 6-7 months (all hot and search/aggregatable)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 1, 2016, 2:59am UTC](https://discuss.elastic.co/t/size-of-index/43065/2 "2016-03-01T02:59:19Z")

</div>

That blog post is pretty old! While some of it is still relevant, be aware that things change over time.

> [@Karthik\_Ramachandran](#):
>
> Case 1 is Great compression where as Case 2 is opposite way. The 500K is a subset for 15 Millon. If my understanding is correct it is because of repetitive terms that come from analyzed field.
> 
> Question 1: Can someone clarify please?

Correct.

> [@Karthik\_Ramachandran](#):
>
> Question 2 : How is it that size is so greater than original text. I tried doing /v1/\_analyze...on analyzed content and it translates to 18 terms

Depends what the document looks like.

> [@Karthik\_Ramachandran](#):
>
> Question 3: Why docs is 5. get cat/indices/v1,v2,v3?v also says 5 as document count, though it is only one. get /v1/count says correctly as 1.

Count includes deleted docs, it could be that.

> [@Karthik\_Ramachandran](#):
>
> Question 4: What is a recommended size of Shard and how many shard we could have? \<=50 GB on a 14GB RAM machine? Is there any logic for computing the same.

That's a larger question not directly answerable by providing a number of shards.  
But you should setup a test that creates a number of indices on the node and see what it can cope with.

---

<div class="post-metadata">

**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\
**Post date:** [March 1, 2016, 6:20am UTC](https://discuss.elastic.co/t/size-of-index/43065/3 "2016-03-01T06:20:05Z")

</div>

Thanks Mark. But for Q3, I didn't delete any documents. I just inserted viz. POST one document and took the metrics.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 1, 2016, 6:28am UTC](https://discuss.elastic.co/t/size-of-index/43065/4 "2016-03-01T06:28:07Z")

</div>

Then you must have indexed more?

---

<div class="post-metadata">

**Author:** ![Youxu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/youxu/32/117375_2.png) [@Youxu](https://discuss.elastic.co/u/Youxu)\
**Post date:** [March 1, 2016, 6:55am UTC](https://discuss.elastic.co/t/size-of-index/43065/5 "2016-03-01T06:55:11Z")

</div>

For the Q3, it is better you post your complete repro steps (with curl commands), this can help others better understand your scenario and identify the root cause easier.

---

<div class="post-metadata">

**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\
**Post date:** [March 3, 2016, 7:45pm UTC](https://discuss.elastic.co/t/size-of-index/43065/6 "2016-03-03T19:45:18Z")

</div>

Below is the sequence of commands I used.

1. Create Index with mappings attached

2. Check for document counts  
get \_cat/indices/test?v  
get \_cat/shards/test?v  
get /test/\_count

3. Add one single document using POST  
POST /test/en/1207407677  
{"DId":"38383838383383838","date":"2015-12-06T07:27:23","From":"TWITTER","Title":"","Link":"[https://twitter.com/test/test/673403345713815552","SourceDomain":"twitter.com","Content":"@sadfasdfasf](https://twitter.com/test/test/673403345713815552%22,%22SourceDomain%22:%22twitter.com%22,%22Content%22:%22@sadfasdfasf) Join us for the event on ABC tech and explore more https:\/\/t.co\/SDDJDJD via https:\/\/t.co\/RUXLEISC","FriendsCount":20543,"FollowersCount":34583,"Score":null}

4. Check the count  
get \_cat/indices/test?v  
get \_cat/shards/test?v  
get /test/\_count

ES Version 2.2.0

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 3, 2016, 9:18pm UTC](https://discuss.elastic.co/t/size-of-index/43065/7 "2016-03-03T21:18:05Z")

</div>

> [@Karthik\_Ramachandran](#):
>
> Create Index with mappings attached

Attached where? I think you may have missed this 🙂

---

<div class="post-metadata">

**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\
**Post date:** [March 3, 2016, 9:20pm UTC](https://discuss.elastic.co/t/size-of-index/43065/8 "2016-03-03T21:20:13Z")

</div>

Hi Mark. I created the mappings representing the POST. But we can report without mapping as well :-).. I should have removed that (1.).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:11pm UTC](https://discuss.elastic.co/t/size-of-index/43065/9 "2017-07-05T23:11:26Z")

</div>