Index Lifecycle Management - Not Deleting Properly

Elastic Stack Elasticsearch
1

I'm experimenting with Index Lifecycle Management in version 6.7.2. Currently just running Locally.

I've set up a pretty simple policy

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "10m",
            "max_size": "50G"
          }
        }
      },
      "warm": {
        "min_age": "12m",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "shrink": {
            "number_of_shards": 1
          }
        }
      },
      "delete": {
        "min_age": "1h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

And a simple template

{
    "index_patterns": ["auditlog*"],
    "settings": {
        "index.lifecycle.name": "auditlog_policy",
        "index.lifecycle.rollover_alias": "auditlog-current"
    },
    "mappings": {
        ...
     }
}

Created my first index, and then started to observe what is happening. What I am finding to be happening. Here are some logs of the events, as well as the current state of indices:

[2019-05-03T14:19:20,241][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [auditlog-000001] creating index, cause [api], templates [auditlog_template], shards [5]/[1], mappings [_doc]
[2019-05-03T14:31:10,344][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [auditlog-000002] creating index, cause [rollover_index], templates [auditlog_template], shards [5]/[1], mappings [_doc]
[2019-05-03T17:04:26,857][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [auditlog-000003] creating index, cause [rollover_index], templates [auditlog_template], shards [5]/[1], mappings [_doc]
[2019-05-03T17:11:10,565][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [shrink-auditlog-000001] creating index, cause [shrink_index], templates [], shards [1]/[1], mappings []
[2019-05-03T17:11:10,606][INFO ][o.e.c.m.MetaDataMappingService] [-fkYLc0] [shrink-auditlog-000001/eygr5L9qS6GrNfrBd-OJCg] create_mapping [_doc]
[2019-05-03T17:21:10,379][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [auditlog-000004] creating index, cause [rollover_index], templates [auditlog_template], shards [5]/[1], mappings [_doc]
[2019-05-03T17:31:10,605][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [shrink-auditlog-000002] creating index, cause [shrink_index], templates [], shards [1]/[1], mappings []
[2019-05-03T17:31:10,643][INFO ][o.e.c.m.MetaDataMappingService] [-fkYLc0] [shrink-auditlog-000002/1fzxc9cLSMexMxdfAYNoXQ] create_mapping [_doc]
[2019-05-03T17:41:10,388][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [auditlog-000005] creating index, cause [rollover_index], templates [auditlog_template], shards [5]/[1], mappings [_doc]
[2019-05-03T17:51:10,679][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [shrink-auditlog-000003] creating index, cause [shrink_index], templates [], shards [1]/[1], mappings []
[2019-05-03T17:51:10,715][INFO ][o.e.c.m.MetaDataMappingService] [-fkYLc0] [shrink-auditlog-000003/wloYR-x7RV2erGVPODIWpg] create_mapping [_doc]
[2019-05-03T18:01:10,393][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [auditlog-000006] creating index, cause [rollover_index], templates [auditlog_template], shards [5]/[1], mappings [_doc]
[2019-05-03T18:11:10,342][INFO ][o.e.c.m.MetaDataDeleteIndexService] [-fkYLc0] [shrink-auditlog-000001/eygr5L9qS6GrNfrBd-OJCg] deleting index
[2019-05-03T18:11:10,764][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [shrink-auditlog-000004] creating index, cause [shrink_index], templates [], shards [1]/[1], mappings []
[2019-05-03T18:11:10,804][INFO ][o.e.c.m.MetaDataMappingService] [-fkYLc0] [shrink-auditlog-000004/MwmWRy1uSUy7V-mfA4x1iQ] create_mapping [_doc]
[2019-05-03T18:21:10,392][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [auditlog-000007] creating index, cause [rollover_index], templates [auditlog_template], shards [5]/[1], mappings [_doc]
[2019-05-03T18:31:10,453][INFO ][o.e.c.m.MetaDataDeleteIndexService] [-fkYLc0] [shrink-auditlog-000002/1fzxc9cLSMexMxdfAYNoXQ] deleting index
[2019-05-03T18:31:10,703][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [shrink-auditlog-000005] creating index, cause [shrink_index], templates [], shards [1]/[1], mappings []
[2019-05-03T18:31:10,740][INFO ][o.e.c.m.MetaDataMappingService] [-fkYLc0] [shrink-auditlog-000005/OH_eGRwKRKW7zIW2K4wn4w] create_mapping [_doc]
[2019-05-03T18:41:10,410][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [auditlog-000008] creating index, cause [rollover_index], templates [auditlog_template], shards [5]/[1], mappings [_doc]
[2019-05-03T18:51:10,376][INFO ][o.e.c.m.MetaDataDeleteIndexService] [-fkYLc0] [shrink-auditlog-000003/wloYR-x7RV2erGVPODIWpg] deleting index
[2019-05-03T18:51:10,675][INFO ][o.e.c.m.MetaDataCreateIndexService] [-fkYLc0] [shrink-auditlog-000006] creating index, cause [shrink_index], templates [], shards [1]/[1], mappings []
[2019-05-03T18:51:10,705][INFO ][o.e.c.m.MetaDataMappingService] [-fkYLc0] [shrink-auditlog-000006/LQVjEnxIRriUIUmXCTGLrA] create_mapping [_doc]

curl -X GET "http://localhost:9200/_cat/indices?pretty"
yellow open shrink-auditlog-000003 wloYR-x7RV2erGVPODIWpg 1 1 0 0  261b  261b
yellow open auditlog-000004        ZLFJgr2TTwiw8D9k0vWG4g 5 1 0 0 1.2kb 1.2kb
yellow open auditlog-000008        ImjIBhF0Sou5-Y_TTUc8Dw 5 1 0 0 1.2kb 1.2kb
yellow open auditlog-000001        _AhOO0lfTOua_oj-zXqhng 5 1 0 0 1.2kb 1.2kb
yellow open auditlog-000005        1cHRiVggRpSmvp5lhJccIg 5 1 0 0 1.2kb 1.2kb
yellow open shrink-auditlog-000004 MwmWRy1uSUy7V-mfA4x1iQ 1 1 0 0  261b  261b
yellow open auditlog-000007        O3FeDMhcQeqMR5DJtIO3EQ 5 1 0 0 1.2kb 1.2kb
yellow open shrink-auditlog-000005 OH_eGRwKRKW7zIW2K4wn4w 1 1 0 0  261b  261b
yellow open auditlog-000006        yIYsAb5_RFSk9L9U7FMVPQ 5 1 0 0 1.2kb 1.2kb
yellow open auditlog-000002        y4wCBAwPRVCgo4PqAtaeow 5 1 0 0 1.2kb 1.2kb
yellow open auditlog-000003        2NNP2c0VSuak6J9CpQtEcQ 5 1 0 0 1.2kb 1.2kb

This doesn't seem to be behaving as i would expect it. The rollover is happening correctly with new indices getting periodically created. The shrink-* indices are then created, but my understanding is that once they are fully created and written, the old index name should start to point to the shrink index, which is not happening, as evident by the fact that all the origin indices still have 5 shards.

Then, an hour after the shrink index is created, it gets deleted, but the origin indices are not getting deleted at all.

Maybe I am not correcting understanding how this works or should be configured? Any help or pointers would be appreciated.

Thanks!

2

ping on this?

3

i think i resolved this myself...

This is only happening because i am running LOCALLY with 1 REPLICA. Because the replica shards are never able to get assigned, indices never get to green with all shards assigned, so the handoff after the shrink phase never actually happens.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.