Elasticsearch indices suddenly much larger


(Kofi) #1

I'm trying to figure out why all of the sudden my index size has increase drastically. Nothing has changed about the type of document being imported and yet over the past 2 months, the index size is from 800mb to 1.4gb whereas before, they would be max 200mb

For example an index with 249,316 documents took 240.3mb of space and currently 95,538 documents are taking 1.1gb. Again, the documents being imported are identical. Anyone know why this is happening? Or ways that I can see what documents/fields are causing this?

Kibana 5.4.3


(Mark Walkom) #2

Can you show us the output of _cat/indices?v?


(Kofi) #3
green  open   logstash-2017.09.19 xptHijfaQl-fWPfXmo9R0g   1   1      33933        17353    227.7mb        113.8mb
yellow open   logstash-2018.03.14 Gq85Er4cQrOMVgY10CqRbw   1   2     152425        11581    240.7mb        240.7mb
yellow open   logstash-2018.03.24 Ig8mHgdbQs6_N0A0rk8V8g   1   2      74743         4128    254.6mb        254.6mb
yellow open   logstash-2018.05.06 mUNhldXLQ7SF65YPtSwnsw   1   2      85849         3336    270.4mb        146.9mb
yellow open   logstash-2017.06.27 6KoQV2M4SWy_UzdjqlDyzg   1   2      79586        64986    271.4mb        271.4mb
yellow open   logstash-2017.07.18 Bjrx_-lvSwqIfwrccu0KiQ   1   2      64548        14560    282.5mb        282.5mb
yellow open   logstash-2017.06.23 ZS1xPzZAQD-NY5JrNN14Xg   1   2     192386        95427    289.2mb        289.2mb
yellow open   logstash-2018.03.08 zRzGST7yTF-2xB2vrb96hQ   1   2      72307         3316      323mb          323mb
yellow open   logstash-2018.03.27 MbWCvmHrTbWxfkmtuH33NA   1   2      53173        19981      349mb          349mb
yellow open   logstash-2017.07.04 Y3AZGgOaQfOK0vFTWXHtdQ   1   2     238774       140311    350.7mb        350.7mb
yellow open   logstash-2017.11.07 bzDZUvFQS_ub6wzt3eiwIg   1   2     154350        42479    355.7mb        355.7mb
yellow open   logstash-2018.04.21 CDJHbHpITJqK3JCYn5OcOg   1   2      73566         1082    360.9mb        360.9mb
yellow open   logstash-2018.03.26 owXNUsuRSyyJVCDfy5Mh-w   1   2      86962        12056    380.7mb        380.7mb
yellow open   logstash-2018.03.25 2iXGDZDKTH6kdAw4Z9q1_g   1   2      76748        22848      384mb          384mb
yellow open   logstash-2018.03.28 k-z4BDH6QoOszHyF5vKHfw   1   2      48729        24948    418.4mb        418.4mb
yellow open   logstash-2018.04.04 fz6H40zgTfW5akCJcZ-OBA   1   2      83034        11926    454.6mb        454.6mb
yellow open   logstash-2018.03.18 XLR7dF9-TSaWD4sfjdYKLg   1   2      56504        33925    492.1mb        492.1mb
yellow open   logstash-2017.06.29 1BJRTRMPRKaVJD1nNkiCgA   1   2     213603        94708      495mb          495mb
yellow open   logstash-2018.04.22 B5dg9Z_dST6lSTBGHL3Xiw   1   2      74335        10364    500.8mb        500.8mb
yellow open   logstash-2018.03.21 XEhx57TPQiyunmQ2qdlHYw   1   2     110342         8153    503.7mb        503.7mb
yellow open   logstash-2018.04.20 SIqcNrAMSO2hsASw5r1iDg   1   2      76142        16587    509.2mb        509.2mb
yellow open   logstash-2018.03.02 1uGZy63iRjyF_0oKp5zfoQ   1   2      93320        43708    535.5mb        535.5mb
yellow open   logstash-2018.03.23 hPCESrEdRDiI8MB_sUhodw   1   2      81933        16972    541.4mb        541.4mb
yellow open   logstash-2018.03.09 nuLRAJ0nTZSq-6ECPBekXg   1   2      73228        36481    546.7mb        546.7mb
yellow open   logstash-2018.03.31 e_R6qOdaQlyqxijSzLCqtA   1   2      95825         5637    566.8mb        566.8mb
yellow open   logstash-2018.04.23 kqOTIGZYS--DMGzzwL6jvQ   1   2      74754        36401    571.8mb        571.8mb
yellow open   logstash-2018.04.07 m1D_8ZkPQMWGZsTiAaqejA   1   2      77643        21243    578.8mb        578.8mb
yellow open   logstash-2018.03.01 laGQzo7WSx6ntXzpOC8Jvg   1   2      74572        73904    593.7mb        593.7mb
yellow open   logstash-2018.04.02 BNhqZqslT1WmVghPv1S7OA   1   2      82741        22673    636.2mb        636.2mb
yellow open   logstash-2018.04.03 wtYKldFvQuKYV4ap7O_d6g   1   2      88797        30889      650mb          650mb
yellow open   logstash-2018.04.09 Lh0_FTZeTDOXIQhMqdWMKw   1   2      77117        27302    656.1mb        656.1mb
yellow open   logstash-2018.04.12 I0_IlTDpSi-RemVhiUXZPg   1   2      89091        25014    673.8mb        673.8mb
yellow open   logstash-2018.03.29 N1FfIIs9TOyT91A8RArCqw   1   2     101694        17296    679.7mb        679.7mb
yellow open   logstash-2018.04.18 SAUWoqFcSmm64vneD_hgPA   1   2      71321        36466    695.9mb        695.9mb
yellow open   logstash-2018.04.01 JGI1BTkwS1GfbXlAItMUmg   1   2      95649        20077    716.2mb        716.2mb
yellow open   logstash-2018.04.06 c5lXLbtyRGeUpCJS6zq37A   1   2      80162        43375    716.7mb        716.7mb
yellow open   logstash-2018.03.06 U7Ex2Z9jQQ6mOIhkCzuVjw   1   2     101215        36367    753.6mb        753.6mb
yellow open   logstash-2018.03.11 0wCaKlOkSiS7JAnpJXoDjA   1   2     135678         7635    758.2mb        758.2mb
yellow open   logstash-2018.04.13 wwPcUyZ8T3KdqXorICg15Q   1   2      89106        23638    763.9mb        763.9mb
yellow open   logstash-2018.03.20 hcSOu2c0Q0ircEe-3pRlvg   1   2     125216        20890    780.4mb        780.4mb
yellow open   logstash-2018.03.03 FfIHmU-wQnS3flBLGAzqMw   1   2     124123        67397    786.5mb        786.5mb
yellow open   logstash-2018.04.16 CetAAJauT0ymCZ6Nx6_P2g   1   2      85636        27619    813.2mb        813.2mb
yellow open   logstash-2018.04.15 vZE1GKOBQcCjYZHCHun9xg   1   2      85831        36358      814mb          814mb
yellow open   logstash-2018.03.30 v6n_0Hg7QZuXyThKh8jq-A   1   2     102190        32898    820.4mb        820.4mb
yellow open   logstash-2018.03.10 13ezyI4-Qd6l1dqlbnMs5w   1   2     139718         6602    834.4mb        834.4mb
yellow open   logstash-2018.03.13 bNfTcH9iTYWRRXWlVOgp4Q   1   2     146066            9    864.4mb        864.4mb
yellow open   logstash-2018.03.04 6LC7mMBJQxCbpVUEeZ-w5g   1   2     123826        40440    891.4mb        891.4mb
yellow open   logstash-2018.04.14 OFcv1X74Qve8t5v3Oj8gYQ   1   2      86235        37356    902.3mb        902.3mb
yellow open   logstash-2018.04.17 RT10h8GfSou36jqHpnG_1Q   1   2      83361        44780    905.6mb        905.6mb
yellow open   logstash-2018.04.10 GWDXBiLOT_GbD4_a6F4Qbg   1   2      92386        36363    906.4mb        906.4mb
yellow open   logstash-2018.03.22 r1HRRCIpSbml5bKlmXLqmQ   1   2     101552        40906    909.7mb        909.7mb
yellow open   logstash-2018.03.12 TLupYj3fQra8nbJhHRfgDg   1   2     143314        11507    929.2mb        929.2mb
yellow open   logstash-2018.04.08 PQ9KxKkhRz6zROghCTzzYQ   1   2      77015        40033    931.3mb        931.3mb
yellow open   logstash-2018.04.05 czaVm9lgRhaYuQnR6k8twg   1   2      89163        40744    965.2mb        965.2mb
yellow open   logstash-2018.04.11 zACABWWHQzOVlft0rRDzCQ   1   2      95538        37630    982.1mb        982.1mb
yellow open   logstash-2018.03.17 2Ud2wH9JSGKqEzyQGPcHOw   1   2     121540        51182   1012.1mb       1012.1mb
yellow open   logstash-2018.03.16 8JD1xsJAQo6fi9KHX1C3-g   1   2     132841        29617   1018.8mb       1018.8mb
yellow open   logstash-2018.03.05 WVMdLJsoTVmQ1rz1RPW87Q   1   2     127459        56619        1gb            1gb
yellow open   logstash-2018.04.19 SeBRNsnSQpyNeOOuXRn7Wg   1   2      75595        40480      1.3gb        678.9mb
yellow open   logstash-2018.03.15 MEMAqnWSTfKIYTbskkd0wA   1   2     139604        63125      1.3gb          1.3gb

(Kofi) #4

@warkolm Bump :smiley:


(Mark Walkom) #5

You should take a look at your mappings and see if there are changes that might explain this.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.