ILM policy: Document count reduced but Index count unchanged

A timeline of events:

  1. At some point, I inadvertently changed max index size changed from 50GB to 1GB.
  2. After some time, I began getting circuit-breaker errors about maxing out the Java heap (31GB on each of the 4 nodes).
  3. Instead of adding another node (which isn't feasible at this time), I decided to reduce the index retention period (as well as set max index size back to 50GB). My thought was that this would reduce the number of shards and thus the memory requirements. The new ILM policy is:
{
  "logstash-policy" : {
    "version" : 19,
    "modified_date" : "2022-06-08T15:10:21.081Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "forcemerge" : {
              "max_num_segments" : 1
            },
            "readonly" : { },
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "delete" : {
          "min_age" : "180d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        },
        "warm" : {
          "min_age" : "90d",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 0,
              "include" : { },
              "exclude" : { },
              "require" : { }
            },
            "forcemerge" : {
              "max_num_segments" : 1
            },
            "readonly" : { },
            "set_priority" : {
              "priority" : 50
            },
            "shrink" : {
              "number_of_shards" : 1
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [
        "ecs-logstash-2022.03.14-000822",
        "ecs-logstash-2022.04.05-001903",
        "ecs-logstash-2022.04.05-001902",
        "ecs-logstash-2022.03.14-000821",
        "ecs-logstash-2022.03.14-000820",
        "ecs-logstash-2022.04.05-001905",
         ...
  1. Soon after changing the ILM policy, I did see the count of documents change from 10B to 5B. However, it's been a few days, and I'm still not seeing the index or shard count reduce. My Java heap usage is still high (~90%), and I'm not exactly sure what's going on at this point. What do I need to do to reduce my index count, and why did the document count drop so much?

Thanks,

--C

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.