# ElasticSearch 2.3 Purge of deleted documents

**URL:** <https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [February 7, 2022, 3:38pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492 "2022-02-07T15:38:20Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [February 7, 2022, 3:38pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/1 "2022-02-07T15:38:20Z")

</div>

I've a legacy Elasticsearch 2.3 index. It is no longer being updated with **new** entries, instead the only modifications are **deletes**. All our new data targets an Elasticsearch 7 cluster. Though we do still read from the ES2 index.

The goal for the legacy 2.3 index is to scale it down. One year ago it was made up of ~9.5 billion **active** documents with ~1 billion **deleted** documents. Now those numbers have changed to ~8 billion **active** documents with ~ 2.5 billion **deleted** documents.

My concern is that the **total** number of documents has not reduced much at all. i.e. It has stayed at ~10.5 billion. This is somewhat of a problem as it is delaying our efforts to scale back the 2.3 cluster.

Over the last year only ~47 million documents were **purged**. ~37 million of which were purged over a ~9 day period.

Reading up on this topic I became aware of the [forcemerge](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-forcemerge.html) API. Is this my best option? Running it against a local (unfortunately ES7) docker cluster I found that it does remove deleted documents when `only_expunge_deletes` is set to `true`.

But there's obviously a big difference between a local docker cluster (with the wrong ES version) and a production index which is still being read.

In short what's my safest option or strategy to tackle this problem and reduce the ES2 disk size?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 7, 2022, 10:50pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/2 "2022-02-07T22:50:44Z")

</div>

Welcome to our community! 😃

What is the output from the `_cluster/stats?pretty&human` API?

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [February 11, 2022, 12:28pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/3 "2022-02-11T12:28:05Z")

</div>

> [@warkolm](#):
>
> \_cluster/stats

Apologies for the delay. Elasticsearch is secured and it took time to get "access". The output of the `_cluster/stats?pretty&human` API is

```auto
{
  "timestamp" : 1644580000432,
  "cluster_name" : "es-2",
  "status" : "green",
  "indices" : {
    "count" : 11,
    "shards" : {
      "total" : 2700,
      "primaries" : 900,
      "replication" : 2.0,
      "index" : {
        "shards" : {
          "min" : 90,
          "max" : 360,
          "avg" : 245.45454545454547
        },
        "primaries" : {
          "min" : 30,
          "max" : 120,
          "avg" : 81.81818181818181
        },
        "replication" : {
          "min" : 2.0,
          "max" : 2.0,
          "avg" : 2.0
        }
      }
    },
    "docs" : {
      "count" : 22774714940,
      "deleted" : 4486222697
    },
    "store" : {
      "size" : "194.8tb",
      "size_in_bytes" : 214196786165418,
      "throttle_time" : "0s",
      "throttle_time_in_millis" : 0
    },
    "fielddata" : {
      "memory_size" : "0b",
      "memory_size_in_bytes" : 0,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size" : "53.9gb",
      "memory_size_in_bytes" : 57882290080,
      "total_count" : 40630798860909,
      "hit_count" : 1065115160391,
      "miss_count" : 39565683700518,
      "cache_size" : 7142906,
      "cache_count" : 7611572316,
      "evictions" : 7604429410
    },
    "completion" : {
      "size" : "0b",
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 108370,
      "memory" : "314.7gb",
      "memory_in_bytes" : 337999262370,
      "terms_memory" : "265gb",
      "terms_memory_in_bytes" : 284574694758,
      "stored_fields_memory" : "49.3gb",
      "stored_fields_memory_in_bytes" : 52946400288,
      "term_vectors_memory" : "0b",
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory" : "29mb",
      "norms_memory_in_bytes" : 30450752,
      "doc_values_memory" : "426.9mb",
      "doc_values_memory_in_bytes" : 447716572,
      "index_writer_memory" : "2.3mb",
      "index_writer_memory_in_bytes" : 2439213,
      "index_writer_max_memory" : "256gb",
      "index_writer_max_memory_in_bytes" : 274925445120,
      "version_map_memory" : "4.6mb",
      "version_map_memory_in_bytes" : 4893886,
      "fixed_bit_set" : "0b",
      "fixed_bit_set_memory_in_bytes" : 0
    },
    "percolate" : {
      "total" : 0,
      "time" : "0s",
      "time_in_millis" : 0,
      "current" : 0,
      "memory_size_in_bytes" : -1,
      "memory_size" : "-1b",
      "queries" : 0
    }
  },
  "nodes" : {
    "count" : {
      "total" : 183,
      "master_only" : 3,
      "data_only" : 180,
      "master_data" : 0,
      "client" : 0
    },
    "versions" : ["2.3.4"],
    "os" : {
      "available_processors" : 1452,
      "allocated_processors" : 1452,
      "mem" : {
        "total" : "81.3gb",
        "total_in_bytes" : 87358369792
      },
      "names" : [ {
        "name" : "Linux",
        "count" : 183
      } ]
    },
    "process" : {
      "cpu" : {
        "percent" : 1413
      },
      "open_file_descriptors" : {
        "min" : 4910,
        "max" : 5986,
        "avg" : 5872
      }
    },
    "jvm" : {
      "max_uptime" : "328.4d",
      "max_uptime_in_millis" : 28374829655,
      "versions" : [ {
        "version" : "1.8.0_312",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "25.312-b07",
        "vm_vendor" : "Private Build",
        "count" : 5
      }, {
        "version" : "1.8.0_292",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "25.292-b10",
        "vm_vendor" : "Private Build",
        "count" : 86
      }, {
        "version" : "1.8.0_282",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "25.282-b08",
        "vm_vendor" : "Private Build",
        "count" : 92
      } ],
      "mem" : {
        "heap_used" : "1.7tb",
        "heap_used_in_bytes" : 1913472249840,
        "heap_max" : "5.3tb",
        "heap_max_in_bytes" : 5907956367360
      },
      "threads" : 17946
    },
    "fs" : {
      "total" : "306.1tb",
      "total_in_bytes" : 336563341922304,
      "free" : "111.2tb",
      "free_in_bytes" : 122334671081472,
      "available" : "95.7tb",
      "available_in_bytes" : 105231601127424,
      "spins" : "true"
    },
    "plugins" : [ {
      "name" : "cloud-aws",
      "version" : "2.3.4",
      "description" : "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.",
      "jvm" : true,
      "classname" : "org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin",
      "isolated" : true,
      "site" : false
    } ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [February 11, 2022, 1:23pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/4 "2022-02-11T13:23:39Z")

</div>

I'll also include a subset of the \_cat/segments API output in case it helps. I'm including 200 lines but the total output is 20,612 lines. The pattern through out is fairly consistent. i.e. Most 4.x gb with more than 10% of the segment made up of deleted documents.

```auto
index shard prirep ip segment generation docs.count docs.deleted size size.memory committed searchable version compound 
this-legacy-index 0 p 170.15.224.249 _1z6e 92246 1033409 924942 4.8gb 8110076 true true 5.5.0 false    
this-legacy-index 0 p 170.15.224.249 _2jgy 118546 1088921 879450 4.7gb 8047115 true true 5.5.0 false    
this-legacy-index 0 p 170.15.224.249 _3i97 163627 978386 971918 4.8gb 8146482 true true 5.5.0 false    
this-legacy-index 0 p 170.15.224.249 _uzpf 1445955 1228826 636928 4.3gb 7353127 true true 5.5.0 false    
this-legacy-index 0 p 170.15.224.249 _1p68o 2854104 1560747 647919 4.7gb 8281536 true true 5.5.0 false    
this-legacy-index 0 p 170.15.224.249 _2kr7d 4327609 1409182 702905 4.7gb 7979983 true true 5.5.0 false    
this-legacy-index 0 p 170.15.224.249 _3gl2s 5812660 1381051 703342 4.7gb 7828301 true true 5.5.0 false    
this-legacy-index 0 p 170.15.224.249 _4956b 7145075 1258540 750513 4.7gb 7613254 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _52g1p 8512189 1292685 765610 4.8gb 7792213 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _5t9u1 9763849 1214460 764898 4.7gb 7530049 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _6kklu 11037522 1293912 761687 4.8gb 7766129 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _7b9u1 12283273 1234451 822640 4.7gb 7769514 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _823ur 13535235 1173378 850844 4.7gb 7691478 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _8rt3v 14734363 1136334 847652 4.7gb 7570359 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _9guml 15902733 1097709 861771 4.6gb 7525942 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _a8def 17186775 1155412 912636 4.7gb 7998695 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ax6nv 18344443 1122010 954425 4.7gb 8019965 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _blluz 19483883 1341634 759827 4.7gb 8132096 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _c8a9r 20541951 1587241 458037 4.7gb 8016551 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _csobj 21493279 1456248 379588 4.7gb 7218389 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ddxb6 22484706 1585290 428290 4.7gb 7857526 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _e0bmp 23529697 1500380 570082 4.7gb 8084158 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _elvuj 24535675 1566915 480922 4.7gb 7955695 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _f8d1c 25584384 1393772 412989 4.2gb 7195938 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _fqst8 26444636 1540712 454695 4.7gb 7913583 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _gbgbs 27408232 1521503 465927 4.7gb 7940801 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _gvkkx 28346865 1523454 469611 4.7gb 7940880 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _hef1s 29226160 1553885 459495 4.7gb 7963464 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _hxnzu 30124218 1563428 476812 4.7gb 8064339 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _igotd 31011745 1558453 460187 4.7gb 7994506 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _izmgz 31895171 1588446 455625 4.7gb 8053783 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _jgxdl 32702457 1584505 447416 4.7gb 8052734 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _jxscn 33489095 1566022 448134 4.7gb 7991055 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _kerdi 34280982 1572012 456498 4.7gb 8068358 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _kwane 35099114 1406018 399933 4.2gb 7165549 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _l9gpu 35713506 1545005 479883 4.7gb 7940216 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _lq5p5 36492377 1594182 439608 4.7gb 8017082 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _m6o1m 37262650 1389830 360789 4.1gb 6972345 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _mluq5 37971149 1565736 470742 4.7gb 8121935 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _n0xi2 38674586 1595061 399011 4.7gb 7926300 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _nmn7c 39687672 1518372 472197 4.7gb 8010125 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _o0t5v 40348579 1596618 397107 4.7gb 7962851 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _o9acn 40744103 1571758 391045 4.7gb 7895370 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _oes4z 41000435 1602880 391976 4.8gb 7981855 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ok87c 41254536 1609468 372221 4.7gb 7922451 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _oq9cs 41535964 1604077 372318 4.7gb 7925655 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ow8di 41814630 1621482 368169 4.8gb 7963246 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _p1hmu 42059910 1504047 333710 4.4gb 7418747 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _p601k 42270392 1656617 318955 4.7gb 7880648 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _p8bkj 42378643 1528732 284416 4.2gb 7152957 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pafda 42476878 1698232 325584 4.7gb 7987388 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pcpgo 42583272 1692114 331992 4.7gb 7898647 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pe86r 42654195 1709993 322691 4.7gb 7947205 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pgjy8 42762752 1692737 313078 4.7gb 7887205 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pj2zd 42880729 1667321 291654 4.7gb 7798307 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pklax 42951129 1709296 298500 4.7gb 7955646 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pmwin 43058975 1690144 306935 4.7gb 7943090 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pou40 43149168 1700059 295965 4.7gb 7943348 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pqxef 43246743 1650518 299734 4.7gb 7765055 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ptccy 43359442 1591914 304739 4.8gb 7743666 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pv1a1 43438393 1672263 301201 4.7gb 7835663 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pxdqo 43547856 1663949 311614 4.8gb 7915486 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _pzno7 43654039 1660578 312869 4.8gb 7884585 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _q1aoc 43730508 1661007 317971 4.8gb 7928777 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _q3xg3 43853331 1656464 327509 4.8gb 7933117 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _q5kgd 43929805 1644799 323622 4.7gb 7825997 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _q7wfd 44038633 1631088 338816 4.8gb 7906414 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qa9h6 44148858 1641073 377973 4.8gb 7962912 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qbs7n 44219795 1608325 397789 4.8gb 7958534 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qe4h2 44328998 1624169 403086 4.8gb 7979981 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qfgq3 44391531 1585538 386275 4.7gb 7927522 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qhq0k 44496884 1566029 409185 4.8gb 7964558 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qiwsp 44552329 1625183 388450 4.8gb 8000795 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qlehq 44668574 1588603 398816 4.7gb 7931916 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qn0kn 44743847 1620963 400373 4.8gb 8000410 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qp91r 44848143 1611461 393173 4.8gb 7987124 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qr5kc 44936940 1602871 386731 4.8gb 7944388 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qsppy 45009718 1614326 384081 4.8gb 7997663 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _quvae 45110246 1615402 375247 4.8gb 7963096 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qw9el 45175197 1629810 374238 4.8gb 8024387 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qyecn 45274919 1648091 369210 4.8gb 7982844 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _qzqvl 45337809 1644931 380057 4.8gb 8069338 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r1zbr 45442071 1598710 378939 4.8gb 7930241 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r3d22 45506522 1633201 387850 4.8gb 8047913 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r5jxx 45608757 1638247 372477 4.8gb 7975111 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r70sx 45677265 1653071 378395 4.8gb 8099495 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r9573 45776271 1636535 376275 4.8gb 8009951 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r95rx 45777021 76050 18370 247.1mb 474117 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r9h1b 45791615 254176 49309 787mb 1980980 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r9htd 45792625 7320 1791 24.1mb 90372 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r9i6g 45793096 7120 1769 23.6mb 89321 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r9lws 45797932 1140 335 3.8mb 26352 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r9m20 45798120 952 265 3.5mb 21818 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r9t1s 45807184 4525 1162 16.7mb 54208 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r9wao 45811392 650 102 2.2mb 17293 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r9xbz 45812735 807 117 2.8mb 19560 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r9ybt 45814025 888 77 3mb 20586 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _r9yon 45814487 1985 230 7.1mb 31813 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra0gm 45816790 311 6 905.5kb 12343 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra0h0 45816804 220 5 912kb 15323 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra0ha 45816814 258 4 895.5kb 11796 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra0hk 45816824 269 12 915.3kb 12714 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra0ie 45816854 2127824 5748 4.7gb 8736725 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra0r6 45817170 122 2 471.3kb 12309 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra236 45818898 110 0 428.6kb 12070 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra2ag 45819160 125 0 382.5kb 11553 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra2aq 45819170 108 0 402.1kb 12344 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra2b0 45819180 75 0 384.6kb 10984 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra2ba 45819190 113 0 407.1kb 11664 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra2bl 45819201 127 0 411.2kb 12735 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra2bm 45819202 1 0 17kb 7585 true true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra2bn 45819203 1 0 17.2kb 6139 false true 5.5.0 true     
this-legacy-index 0 p 170.15.224.249 _ra2bo 45819204 1 0 18.3kb 6139 false true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _1z6e 92246 1033409 924942 4.8gb 8110076 true true 5.5.0 false    
this-legacy-index 0 r 170.15.215.216 _2jgy 118546 1088921 879450 4.7gb 8047115 true true 5.5.0 false    
this-legacy-index 0 r 170.15.215.216 _3i97 163627 978386 971918 4.8gb 8146482 true true 5.5.0 false    
this-legacy-index 0 r 170.15.215.216 _uzpf 1445955 1228826 636928 4.3gb 7353127 true true 5.5.0 false    
this-legacy-index 0 r 170.15.215.216 _1p68o 2854104 1560747 647919 4.7gb 8281536 true true 5.5.0 false    
this-legacy-index 0 r 170.15.215.216 _2kr7d 4327609 1409182 702905 4.7gb 7979983 true true 5.5.0 false    
this-legacy-index 0 r 170.15.215.216 _3gl2s 5812660 1381051 703342 4.7gb 7828301 true true 5.5.0 false    
this-legacy-index 0 r 170.15.215.216 _4956b 7145075 1258540 750513 4.7gb 7613254 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _52g1p 8512189 1292685 765610 4.8gb 7792213 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _5t9u1 9763849 1214460 764898 4.7gb 7530049 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _6kklu 11037522 1293912 761687 4.8gb 7766129 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _7b9u1 12283273 1234451 822640 4.7gb 7769514 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _823ur 13535235 1173378 850844 4.7gb 7691478 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _8rt3v 14734363 1136334 847652 4.7gb 7570359 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _9guml 15902733 1097709 861771 4.6gb 7525942 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _a8def 17186775 1155412 912636 4.7gb 7998695 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _ax6nv 18344443 1122010 954425 4.7gb 8019965 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _blluz 19483883 1341634 759827 4.7gb 8132096 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _c8a9r 20541951 1587241 458037 4.7gb 8016551 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _csobj 21493279 1456248 379588 4.7gb 7218389 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _ddxb6 22484706 1585290 428290 4.7gb 7857526 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _e0bmp 23529697 1500380 570082 4.7gb 8084158 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _elvuj 24535675 1566915 480922 4.7gb 7955695 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _f8d1c 25584384 1393772 412989 4.2gb 7195938 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _fqst8 26444636 1540712 454695 4.7gb 7913583 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _gbgbs 27408232 1521503 465927 4.7gb 7940801 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _gvkkx 28346865 1523454 469611 4.7gb 7940880 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _hef1s 29226160 1553885 459495 4.7gb 7963464 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _hxnzu 30124218 1563428 476812 4.7gb 8064339 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _igotd 31011745 1558453 460187 4.7gb 7994506 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _izmgz 31895171 1588446 455625 4.7gb 8053783 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _jgxdl 32702457 1584505 447416 4.7gb 8052734 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _jxscn 33489095 1566022 448134 4.7gb 7991055 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _kerdi 34280982 1572012 456498 4.7gb 8068358 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _kwane 35099114 1406018 399933 4.2gb 7165549 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _l9gpu 35713506 1545005 479883 4.7gb 7940216 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _lq5p5 36492377 1594182 439608 4.7gb 8017082 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _m6o1m 37262650 1389830 360789 4.1gb 6972345 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _mluq5 37971149 1565736 470742 4.7gb 8121935 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _n0xi2 38674586 1595061 399011 4.7gb 7926300 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _nmn7c 39687672 1518372 472197 4.7gb 8010125 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _o0t5v 40348579 1596618 397107 4.7gb 7962851 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _o9acn 40744103 1571758 391045 4.7gb 7895370 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _oes4z 41000435 1602880 391976 4.8gb 7981855 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _ok87c 41254536 1609468 372221 4.7gb 7922451 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _oq9cs 41535964 1604077 372318 4.7gb 7925655 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _ow8di 41814630 1621482 368169 4.8gb 7963246 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _p1hmu 42059910 1504047 333710 4.4gb 7418747 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _p601k 42270392 1656617 318955 4.7gb 7880648 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _p8bkj 42378643 1528732 284416 4.2gb 7152957 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pafda 42476878 1698232 325584 4.7gb 7987388 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pcpgo 42583272 1692114 331992 4.7gb 7898647 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pe86r 42654195 1709993 322691 4.7gb 7947205 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pgjy8 42762752 1692737 313078 4.7gb 7887205 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pj2zd 42880729 1667321 291654 4.7gb 7798307 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pklax 42951129 1709296 298500 4.7gb 7955646 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pmwin 43058975 1690144 306935 4.7gb 7943090 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pou40 43149168 1700059 295965 4.7gb 7943348 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pqxef 43246743 1650518 299734 4.7gb 7765055 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _ptccy 43359442 1591914 304739 4.8gb 7743666 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pv1a1 43438393 1672263 301201 4.7gb 7835663 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pxdqo 43547856 1663949 311614 4.8gb 7915486 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _pzno7 43654039 1660578 312869 4.8gb 7884585 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _q1aoc 43730508 1661007 317971 4.8gb 7928777 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _q3xg3 43853331 1656464 327509 4.8gb 7933117 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _q5kgd 43929805 1644799 323622 4.7gb 7825997 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _q7wfd 44038633 1631088 338816 4.8gb 7906414 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qa9h6 44148858 1641073 377973 4.8gb 7962912 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qbs7n 44219795 1608325 397789 4.8gb 7958534 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qe4h2 44328998 1624169 403086 4.8gb 7979981 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qfgq3 44391531 1585538 386275 4.7gb 7927522 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qhq0k 44496884 1566029 409185 4.8gb 7964558 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qiwsp 44552329 1625183 388450 4.8gb 8000795 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qlehq 44668574 1588603 398816 4.7gb 7931916 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qn0kn 44743847 1620963 400373 4.8gb 8000410 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qp91r 44848143 1611461 393173 4.8gb 7987124 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qr5kc 44936940 1602871 386731 4.8gb 7944388 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qsppy 45009718 1614326 384081 4.8gb 7997663 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _quvae 45110246 1615402 375247 4.8gb 7963096 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qw9el 45175197 1629810 374238 4.8gb 8024387 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qyecn 45274919 1648091 369210 4.8gb 7982844 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _qzqvl 45337809 1644931 380057 4.8gb 8069338 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _r1zbr 45442071 1598710 378939 4.8gb 7930241 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _r3d22 45506522 1633201 387850 4.8gb 8047913 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _r5jxx 45608757 1638247 372477 4.8gb 7975111 true true 5.5.0 true     
this-legacy-index 0 r 170.15.215.216 _r70sx 45677265 1653071 378395 4.8gb 8099495 true true 5.5.0 true     

```

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [February 11, 2022, 2:28pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/5 "2022-02-11T14:28:20Z")

</div>

I should mention there are indexes other than `this-legacy-index` on the ES2 cluster. I'm focusing on `this-legacy-index` as it is by far the largest and oldest. The other indexes have a similar segmentation, though there might be slightly less 4.x gb segments.

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [February 11, 2022, 4:46pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/6 "2022-02-11T16:46:17Z")

</div>

Reading over the forcemerge documentation I noticed that the [8.0 version](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html) mentions [the `force_merge` threadpool](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html). There is no such reference in the [2.3 version](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/indices-forcemerge.html) nor is it mentioned in the [2.3 threadpool docs](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-threadpool.html).

Any insights into what this means? Does 2.3 default to a single thread? Or does it use any available thread from a common pool?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 13, 2022, 10:30pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/7 "2022-02-13T22:30:12Z")

</div>

I don't know sorry, you'll probably struggle to get an answer on that given the age of 2.X.

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [February 14, 2022, 6:53pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/8 "2022-02-14T18:53:12Z")

</div>

In terms of 2.3 that's fair enough as it's no longer supported. Have you any advice on the original question?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 15, 2022, 12:12am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/9 "2022-02-15T00:12:40Z")

</div>

I'd try running a force merge on the index, but you might want to do that in a quiet time, as they were not as well managed by the cluster as they are with newer versions.

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [February 28, 2022, 10:40am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/10 "2022-02-28T10:40:10Z")

</div>

A follow up question. We ran a force merge over the weekend. This has caused most our deleted data to be removed. Namely ~2.5 billion documents. That's great.

I can see it also dropped our segment count from ~20k down to ~2k. But the downside is that we now have 110 segments sized with more than 100 GB of data. For performance reasons we've tried to keep our segments below 5 GB.

Any thoughts or recommendations on how we can re-balance the segments?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 28, 2022, 9:57pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/11 "2022-02-28T21:57:27Z")

</div>

You should really be focussing on shards, not segments.

You can run a force merge and specify a number of segments, I am not you can go _up_ though.

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [March 3, 2022, 12:05pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/12 "2022-03-03T12:05:34Z")

</div>

I focused on segments as the advise was to keep them below 5 GB. But I'm not sure how strictly that needs to be followed. Monitoring the legacy index after the forceMerge I see the search duration has increased from peaks of ~700ms up to peaks of ~1.2s. There's also an increase in heap usage from ~2 GB to ~3.3 GB. But that seems acceptable.

For shards we focus on keeping them below ~50 GB.

The goal for the legacy index is to slowly reduce it. As such we've also kicked off a node reduction. i.e. Going from ~180 to ~150 nodes. This could also be contributing to the above.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 3, 2022, 9:36pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/13 "2022-03-03T21:36:53Z")

</div>

> [@Shane\_Gannon](#):
>
> I focused on segments as the advise was to keep them below 5 GB

Advice from who exactly?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 3, 2022, 9:38pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/14 "2022-03-03T21:38:12Z")

</div>

> [@Shane\_Gannon](#):
>
> Going from ~180 to ~150 nodes

Honestly, upgrading will get you much further.  
As a high level example, not taking into account a tonne of things, 7.X should handle those 2700 shards on 4 nodes. 2.X does not manage shards efficiently.

---

<div class="post-metadata">

**Author:** ![Shane\_Gannon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_gannon/32/101481_2.png) [@Shane\_Gannon](https://discuss.elastic.co/u/Shane_Gannon)\
**Post date:** [March 4, 2022, 9:37am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/15 "2022-03-04T09:37:21Z")

</div>

> [@warkolm](#):
>
> Advice from who exactly?

Before my time, a contractor was hired to give advice on the operation of Elasticsearch. One of the take-away points was to keep the segment size below 5 GB. This point stuck in the "team memory".

While that is not exactly reliable, I noticed Elasticsearch itself, before the forceMerge, kept the segment size below 5 GB. e.g. 4.7 GB, 4.8 GB, etc. I also came across blogs like [this](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents) that mention

> A maximum sized segment (default: 5 GB) will only be eligible for merging once it accumulates 50% deletions

> [@warkolm](#):
>
> Honestly, upgrading will get you much further.

I'd be in favour of migrating the old ES2 data to our ES7 cluster. But it was not my call to make, and the conclusion the team reached, is that it'd take too long and be too disruptive.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 1, 2022, 9:37am UTC](https://discuss.elastic.co/t/elasticsearch-2-3-purge-of-deleted-documents/296492/16 "2022-04-01T09:37:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.