# Index Size explosion (17 GB -\> 840 GB) **URL:** \ **Category:** Elasticsearch\ **Created:** [March 25, 2013, 11:13am UTC](https://discuss.elastic.co/t/index-size-explosion-17-gb-840-gb/11290 "2013-03-25T11:13:22Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [March 25, 2013, 11:13am UTC](https://discuss.elastic.co/t/index-size-explosion-17-gb-840-gb/11290/1 "2013-03-25T11:13:22Z")
Hi , I had an index which was initially of 17 GB. It had a single shard and ran in a single machine. Last week i migrated the data to a 4 shard index with the same mapping. These 4 shards are distributed among 4 machines. After migration , i ran a script which updates one of the field of every feed. Now after all the migration and updation , size of index is around 840 GB together each of the shard having around 250 GB of data. I am not able to comprehend hat happened. Kindly shed some light on this issue. Thanks Vineeth -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [March 25, 2013, 11:42am UTC](https://discuss.elastic.co/t/index-size-explosion-17-gb-840-gb/11290/2 "2013-03-25T11:42:17Z")
can you give us more insight in what your script does? On Monday, March 25, 2013 12:13:22 PM UTC+1, Vineeth Mohan wrote: > Hi , > > I had an index which was initially of 17 GB. It had a single shard and ran > in a single machine. > Last week i migrated the data to a 4 shard index with the same mapping. > These 4 shards are distributed among 4 machines. > After migration , i ran a script which updates one of the field of every > feed. > Now after all the migration and updation , size of index is around 840 GB > together each of the shard having around 250 GB of data. > > I am not able to comprehend hat happened. > Kindly shed some light on this issue. > > Thanks > Vineeth -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [March 25, 2013, 12:21pm UTC](https://discuss.elastic.co/t/index-size-explosion-17-gb-840-gb/11290/3 "2013-03-25T12:21:34Z")
Adding some more info. The number of replica is 0. Also please find the before and after images of the head plugin attached. Kindly note that the number of feeds is same for both. Also the operation i did on the index after migration was just update requests on all the feeds. Thanks Vineeth On Mon, Mar 25, 2013 at 4:43 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > Hi , > > I had an index which was initially of 17 GB. It had a single shard and ran > in a single machine. > Last week i migrated the data to a 4 shard index with the same mapping. > These 4 shards are distributed among 4 machines. > After migration , i ran a script which updates one of the field of every > feed. > Now after all the migration and updation , size of index is around 840 GB > together each of the shard having around 250 GB of data. > > I am not able to comprehend hat happened. > Kindly shed some light on this issue. > > Thanks > Vineeth -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [March 25, 2013, 12:22pm UTC](https://discuss.elastic.co/t/index-size-explosion-17-gb-840-gb/11290/4 "2013-03-25T12:22:24Z")
Adding attachments. PFA Thanks Vineeth On Mon, Mar 25, 2013 at 5:51 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > Adding some more info. > The number of replica is 0. > > Also please find the before and after images of the head plugin attached. > Kindly note that the number of feeds is same for both. > Also the operation i did on the index after migration was just update > requests on all the feeds. > > Thanks > Vineeth > > On Mon, Mar 25, 2013 at 4:43 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > > > Hi , > > > > I had an index which was initially of 17 GB. It had a single shard and > > ran in a single machine. > > Last week i migrated the data to a 4 shard index with the same mapping. > > These 4 shards are distributed among 4 machines. > > After migration , i ran a script which updates one of the field of every > > feed. > > Now after all the migration and updation , size of index is around 840 GB > > together each of the shard having around 250 GB of data. > > > > I am not able to comprehend hat happened. > > Kindly shed some light on this issue. > > > > Thanks > > Vineeth -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [March 25, 2013, 1:59pm UTC](https://discuss.elastic.co/t/index-size-explosion-17-gb-840-gb/11290/5 "2013-03-25T13:59:45Z")
File size dump of the previus and present index Previous - [gist:5237294 · GitHub](https://gist.github.com/Vineeth-Mohan/5237294) (Single shard , single machine) Present - [gist:5237236 · GitHub](https://gist.github.com/Vineeth-Mohan/5237236) (4 shard , 4 machine) On Mon, Mar 25, 2013 at 5:52 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > Adding attachments. > > PFA > > Thanks > Vineeth > > On Mon, Mar 25, 2013 at 5:51 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > > > Adding some more info. > > The number of replica is 0. > > > > Also please find the before and after images of the head plugin attached. > > Kindly note that the number of feeds is same for both. > > Also the operation i did on the index after migration was just update > > requests on all the feeds. > > > > Thanks > > Vineeth > > > > On Mon, Mar 25, 2013 at 4:43 PM, Vineeth Mohan \<[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) > > > > > wrote: > > > > > Hi , > > > > > > I had an index which was initially of 17 GB. It had a single shard and > > > ran in a single machine. > > > Last week i migrated the data to a 4 shard index with the same mapping. > > > These 4 shards are distributed among 4 machines. > > > After migration , i ran a script which updates one of the field of every > > > feed. > > > Now after all the migration and updation , size of index is around 840 > > > GB together each of the shard having around 250 GB of data. > > > > > > I am not able to comprehend hat happened. > > > Kindly shed some light on this issue. > > > > > > Thanks > > > Vineeth -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [March 25, 2013, 3:40pm UTC](https://discuss.elastic.co/t/index-size-explosion-17-gb-840-gb/11290/6 "2013-03-25T15:40:24Z")
I would guess that the reason for this change in shard sizes is dramatic increase in the average size of the \_source field of your documents. I would suggest checking sources for documents before update and after update to see what changed. Maybe update process didn't go as expected. On Monday, March 25, 2013 9:59:45 AM UTC-4, Vineeth Mohan wrote: > File size dump of the previus and present index > > Previous - [gist:5237294 · GitHub](https://gist.github.com/Vineeth-Mohan/5237294) (Single shard , > single machine) > Present - [gist:5237236 · GitHub](https://gist.github.com/Vineeth-Mohan/5237236) (4 shard , 4 > machine) > > On Mon, Mar 25, 2013 at 5:52 PM, Vineeth Mohan \<[vineet...@algotree.com](mailto:vineet...@algotree.com)\ > > > wrote: > > > Adding attachments. > > > > PFA > > > > Thanks > > Vineeth > > > > On Mon, Mar 25, 2013 at 5:51 PM, Vineeth Mohan \<[vineet...@algotree.com](mailto:vineet...@algotree.com)\ > > > > > wrote: > > > > > Adding some more info. > > > The number of replica is 0. > > > > > > Also please find the before and after images of the head plugin attached. > > > Kindly note that the number of feeds is same for both. > > > Also the operation i did on the index after migration was just update > > > requests on all the feeds. > > > > > > Thanks > > > Vineeth > > > > > > On Mon, Mar 25, 2013 at 4:43 PM, Vineeth Mohan \<[vineet...@algotree.com](mailto:vineet...@algotree.com)\ > > > > > > > wrote: > > > > > > > Hi , > > > > > > > > I had an index which was initially of 17 GB. It had a single shard and > > > > ran in a single machine. > > > > Last week i migrated the data to a 4 shard index with the same mapping. > > > > These 4 shards are distributed among 4 machines. > > > > After migration , i ran a script which updates one of the field of > > > > every feed. > > > > Now after all the migration and updation , size of index is around 840 > > > > GB together each of the shard having around 250 GB of data. > > > > > > > > I am not able to comprehend hat happened. > > > > Kindly shed some light on this issue. > > > > > > > > Thanks > > > > Vineeth -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [March 25, 2013, 5:46pm UTC](https://discuss.elastic.co/t/index-size-explosion-17-gb-840-gb/11290/7 "2013-03-25T17:46:48Z")
Not saying this happened to you, but I've had bugs in update scripts before that recursively include the source in the update. So when a document is updated, I accidentally include the old \_source as a field. The next time the doc is updated, the \_source is again included (which now includes two copies of the old source), etc etc. If you do that enough, the size quickly spirals out of control. Definitely check your script to make sure it is doing what you think it is. -Zach On Monday, March 25, 2013 9:59:45 AM UTC-4, Vineeth Mohan wrote: > File size dump of the previus and present index > > Previous - [gist:5237294 · GitHub](https://gist.github.com/Vineeth-Mohan/5237294) (Single shard , > single machine) > Present - [gist:5237236 · GitHub](https://gist.github.com/Vineeth-Mohan/5237236) (4 shard , 4 > machine) > > On Mon, Mar 25, 2013 at 5:52 PM, Vineeth Mohan \<[vineet...@algotree.com](mailto:vineet...@algotree.com)\ > > > wrote: > > > Adding attachments. > > > > PFA > > > > Thanks > > Vineeth > > > > On Mon, Mar 25, 2013 at 5:51 PM, Vineeth Mohan \<[vineet...@algotree.com](mailto:vineet...@algotree.com)\ > > > > > wrote: > > > > > Adding some more info. > > > The number of replica is 0. > > > > > > Also please find the before and after images of the head plugin attached. > > > Kindly note that the number of feeds is same for both. > > > Also the operation i did on the index after migration was just update > > > requests on all the feeds. > > > > > > Thanks > > > Vineeth > > > > > > On Mon, Mar 25, 2013 at 4:43 PM, Vineeth Mohan \<[vineet...@algotree.com](mailto:vineet...@algotree.com)\ > > > > > > > wrote: > > > > > > > Hi , > > > > > > > > I had an index which was initially of 17 GB. It had a single shard and > > > > ran in a single machine. > > > > Last week i migrated the data to a 4 shard index with the same mapping. > > > > These 4 shards are distributed among 4 machines. > > > > After migration , i ran a script which updates one of the field of > > > > every feed. > > > > Now after all the migration and updation , size of index is around 840 > > > > GB together each of the shard having around 250 GB of data. > > > > > > > > I am not able to comprehend hat happened. > > > > Kindly shed some light on this issue. > > > > > > > > Thanks > > > > Vineeth -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [March 26, 2013, 12:03am UTC](https://discuss.elastic.co/t/index-size-explosion-17-gb-840-gb/11290/8 "2013-03-26T00:03:00Z")
@Zach - Yeah dude , that is what happened to me also. I expected update script to replace a specific field like it do if its a single value field. But if the field is a complex associated variable with hash and arrays , it is actually merging the field with the old one. Hence the size explosion. Thanks @igor for all your help. Thanks Vineeth On Mon, Mar 25, 2013 at 11:16 PM, Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com)wrote: > Not saying this happened to you, but I've had bugs in update scripts > before that recursively include the source in the update. So when a > document is updated, I accidentally include the old \_source as a field. > The next time the doc is updated, the \_source is again included (which now > includes two copies of the old source), etc etc. > > If you do that enough, the size quickly spirals out of control. > Definitely check your script to make sure it is doing what you think it > is. > > -Zach > > On Monday, March 25, 2013 9:59:45 AM UTC-4, Vineeth Mohan wrote: > > > File size dump of the previus and present index > > > > Previous - [https://gist.github.com/\*\*Vineeth-Mohan/5237294](https://gist.github.com/**Vineeth-Mohan/5237294)[https://gist.github.com/Vineeth-Mohan/5237294](https://gist.github.com/Vineeth-Mohan/5237294)(Single shard , single machine) > > Present - [https://gist.github.com/\*\*Vineeth-Mohan/5237236](https://gist.github.com/**Vineeth-Mohan/5237236)[https://gist.github.com/Vineeth-Mohan/5237236](https://gist.github.com/Vineeth-Mohan/5237236)(4 shard , 4 machine) > > > > On Mon, Mar 25, 2013 at 5:52 PM, Vineeth Mohan [vineet...@algotree.com](mailto:vineet...@algotree.com)wrote: > > > > > Adding attachments. > > > > > > PFA > > > > > > Thanks > > > Vineeth > > > > > > On Mon, Mar 25, 2013 at 5:51 PM, Vineeth Mohan [vineet...@algotree.com](mailto:vineet...@algotree.com)wrote: > > > > > > > Adding some more info. > > > > The number of replica is 0. > > > > > > > > Also please find the before and after images of the head plugin > > > > attached. > > > > Kindly note that the number of feeds is same for both. > > > > Also the operation i did on the index after migration was just update > > > > requests on all the feeds. > > > > > > > > Thanks > > > > Vineeth > > > > > > > > On Mon, Mar 25, 2013 at 4:43 PM, Vineeth Mohan [vineet...@algotree.com](mailto:vineet...@algotree.com)wrote: > > > > > > > > > Hi , > > > > > > > > > > I had an index which was initially of 17 GB. It had a single shard and > > > > > ran in a single machine. > > > > > Last week i migrated the data to a 4 shard index with the same > > > > > mapping. These 4 shards are distributed among 4 machines. > > > > > After migration , i ran a script which updates one of the field of > > > > > every feed. > > > > > Now after all the migration and updation , size of index is around 840 > > > > > GB together each of the shard having around 250 GB of data. > > > > > > > > > > I am not able to comprehend hat happened. > > > > > Kindly shed some light on this issue. > > > > > > > > > > Thanks > > > > > Vineeth > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:44am UTC](https://discuss.elastic.co/t/index-size-explosion-17-gb-840-gb/11290/9 "2017-07-06T02:44:30Z")