# Size of index is increasing abnormally? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 21, 2016, 10:43am UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024 "2016-04-21T10:43:10Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![shivam\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivam_singh/32/4225_2.png) [@shivam\_singh](https://discuss.elastic.co/u/shivam_singh)\ **Post date:** [April 21, 2016, 10:43am UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/1 "2016-04-21T10:43:11Z")
Hi, The size of indexes in Elasticsearch is Abnormally increasing. What can be the issue? As u can see in the uploaded snapshot it increased from 1 Gb to 25 GB. Nuber of events came through as shown in kibana UI. 28,404,187 Today 794,774 yesterday 831,019 Day before yesterday ![](https://us1.discourse-cdn.com/elastic/original/2X/2/201038bbe53d2b523536861ba79221f4c7770fad.PNG) ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 21, 2016, 11:30am UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/2 "2016-04-21T11:30:05Z")
Do you have any indication of the data source's sizes for these days? Is there anything to suggest the elasticsearch index sizes are not commensurate with the upstream data sources for these days? Cheers Mark ---
**Author:** ![shivam\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivam_singh/32/4225_2.png) [@shivam\_singh](https://discuss.elastic.co/u/shivam_singh)\ **Post date:** [April 21, 2016, 12:03pm UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/3 "2016-04-21T12:03:53Z")
I have some numbers for sizes of log files for around last 6-8 days. Size of all the log files that i am parsing to logstash and storing into elasticsearch add up to 600 MB approx. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 21, 2016, 12:05pm UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/4 "2016-04-21T12:05:56Z")
And individually do these log files' sizes tally with what you see in index sizes? Is there a constant ratio in the size comparisons? ---
**Author:** ![shivam\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivam_singh/32/4225_2.png) [@shivam\_singh](https://discuss.elastic.co/u/shivam_singh)\ **Post date:** [April 21, 2016, 12:20pm UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/5 "2016-04-21T12:20:41Z")
Logs are coming from some 15 different servers, thus the log files are distributed. I don't know how to compare the individual file size with respect to the corresponding indexes. Anyway, even if we take average 600 MB for 6 Days. Around 100 MB per day. That is nowhere near 20 GB (index size). Also, if we compare the index size from 2 days back its only 1 Gb. Since, this environment had no update/change since Monday, thr is no way logging can increase to such an extent. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 21, 2016, 12:31pm UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/6 "2016-04-21T12:31:46Z")
The number of documents in the indices has increased dramatically as well, but this increase appear roughly proportional to the increase in index size. If you are logging which host the data is coming from, run an aggregation, e.g. using Kibana, comparing the distribution of the new data to the old to see if there is anything that stands out. ---
**Author:** ![shivam\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivam_singh/32/4225_2.png) [@shivam\_singh](https://discuss.elastic.co/u/shivam_singh)\ **Post date:** [April 25, 2016, 8:30am UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/7 "2016-04-25T08:30:26Z")
Hey, I am attaching few snapshots please go through them for better understanding. As you can see from the snapshots the total events that came through on 22nd were more than the total events that came through on 21st and still the size of index on 21st is greater than size of index on 22nd (35GB vs 3.13 GB). Even if we consider the events on both days to be approximately same the size of indexes is no way comparable. Snapshot from 21st 00.00 - 22nd 00.00 ![](https://us1.discourse-cdn.com/elastic/original/2X/e/e5329fed652eeed4aff65625b4f82c77531558db.PNG) Snapshot from 22st 00.00 - 23nd 00.00 ![](https://us1.discourse-cdn.com/elastic/original/2X/e/efb800e2154aa04047f9535438b67fe90800820d.PNG) Kopf index size ![](https://us1.discourse-cdn.com/elastic/original/2X/7/732ab94b82850b83bcc9c6e2f2c0afa27b83afc7.PNG) Thanks ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 25, 2016, 8:58am UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/8 "2016-04-25T08:58:03Z")
The barchart axes aren't labelled so it's hard to tell what the bars represent (I'm assuming Y axis is number of docs?) Also I assume the number of bars shown is only the first 10. It is quite possible there is a long tail of millions of other smaller bars that together add up to a big number. I don't know what field is represented on the X-axis - a multi-valued field would mean each doc may be accounted for by more than one bar. Another point of confusion is how the numbers of docs on the bar chart (at least ~90m docs?) vs those reported by Kopf (4m?). ---
**Author:** ![shivam\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivam_singh/32/4225_2.png) [@shivam\_singh](https://discuss.elastic.co/u/shivam_singh)\ **Post date:** [April 25, 2016, 10:08am UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/9 "2016-04-25T10:08:55Z")
Sorry, on y-axis it number of events and on x axis its the hosts from which the logs are coming from. To give you a more clear picture here is the sum of all events that came through including all the hosts from which logs are coming from (26 different hosts/servers). 21St - ![](https://us1.discourse-cdn.com/elastic/original/2X/f/fcd028b6fb8ca3b65a895caf9bc282650cfb20c1.png) 22nd - ![](https://us1.discourse-cdn.com/elastic/original/2X/7/727a363ca4b1ffb9abb0c75d6642f01d184bb751.png) To yr question "Another point of confusion is how the numbers of docs on the bar chart (at least ~90m docs?) vs those reported by Kopf (4m?)." Its not the count of docs its total events (sum of all log events on particular date ranges). ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 26, 2016, 7:25am UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/10 "2016-04-26T07:25:29Z")
Is an event the same thing as a doc? Need to understand relationship between these two. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 27, 2016, 6:59am UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/11 "2016-04-27T06:59:32Z")
Something is strange here. In Kopf it looks like your `logstash-2016-04-21` index has 20925958 documents taking up 33.54GB on disk. In your Kibana bar charts you are however reporting over 30 million hits for the same period. This does not add up. It is even more off for the `logstash-2016.04.22` index, which according to Kopf only has a bit over 4 million records, yet shows more then 30 million hits in Kibana. Have you selected the time period covered in Kibana correctly? How did you create the Kibana visualisations? ---
**Author:** ![shivam\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivam_singh/32/4225_2.png) [@shivam\_singh](https://discuss.elastic.co/u/shivam_singh)\ **Post date:** [April 28, 2016, 8:22am UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/12 "2016-04-28T08:22:09Z")
Hey Guys, Thanks for the support. I really appreciate the help. I guess I have found the problem. The root cause for this problem is not elasticsearch, it is some new logging that is introduced in our software (I was not informed about this change) and is creating logs abnormally, thus the size of index is increasing abnormally. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 28, 2016, 1:46pm UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/13 "2016-04-28T13:46:57Z")
Thanks for putting an end to the mystery 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:55pm UTC](https://discuss.elastic.co/t/size-of-index-is-increasing-abnormally/48024/14 "2017-07-05T22:55:32Z")