ILM policy not working for all indexes

[SergeSpinoza]

Hi! At some point ILM stopped working for all indexes.

For example - ILM policy for index nginx-*:

GET /_ilm/policy/nginx_index_delete:

{
  "nginx_index_delete": {
    "policy": {
      "phases": {
        "hot": {
          "actions": {
            "set_priority": {
              "priority": 100
            }
          },
          "min_age": "0ms"
        },
        "delete": {
          "actions": {
            "delete": {}
          },
          "min_age": "16d"
        }
      }
    },
    "modified_date": "2020-01-22T10:06:20.292Z",
    "version": 2
  }
}

But old indexes of nginx-* do not deleted automatically.

GET nginx-2020.01.11/_ilm/explain:

{
  "indices": {
    "nginx-2020.01.11": {
      "index": "nginx-2020.01.11",
      "managed": true,
      "phase_execution": {
        "policy": "nginx_index_delete",
        "modified_date_in_millis": 1574062458159,
        "phase_definition": {
          "actions": {
            "set_priority": {
              "priority": 100
            }
          },
          "min_age": "0ms"
        },
        "version": 1
      },
      "action_time_millis": 1578700802422,
      "step_time_millis": 1578700802422,
      "phase_time_millis": 1578700801973,
      "phase": "hot",
      "step": "complete",
      "lifecycle_date_millis": 1578700801243,
      "policy": "nginx_index_delete",
      "action": "complete"
    }
  }
}

GET _template/nginx:

{
  "nginx": {
    "mappings": {
      "doc": {
        "properties": {
          "request_time": {
            "type": "float"
          },
          "upstream_port": {
            "type": "float"
          },
          "geoip": {
            "dynamic": true,
            "properties": {
              "latitude": {
                "type": "half_float"
              },
              "ip": {
                "type": "ip"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "half_float"
              }
            }
          },
          "@timestamp": {
            "type": "date"
          },
          "bytes": {
            "type": "float"
          },
          "upstream_response_time": {
            "type": "float"
          },
          "upstream_addr": {
            "type": "ip"
          },
          "@version": {
            "type": "keyword"
          },
          "response": {
            "type": "float"
          },
          "clientip": {
            "type": "ip"
          }
        },
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "mapping": {
                "norms": false,
                "type": "text"
              },
              "match_mapping_type": "string"
            }
          },
          {
            "string_fields": {
              "match_mapping_type": "string",
              "mapping": {
                "norms": false,
                "fields": {
                  "keyword": {
                    "ignore_above": 256,
                    "type": "keyword"
                  }
                },
                "type": "text"
              },
              "match": "*"
            }
          }
        ]
      }
    },
    "settings": {
      "index": {
        "number_of_replicas": "1",
        "number_of_shards": "1",
        "lifecycle": {
          "name": "nginx_index_delete"
        },
        "refresh_interval": "5s"
      }
    },
    "version": 60001,
    "index_patterns": [
      "nginx-*"
    ],
    "order": 0,
    "aliases": {}
  }
}

What can this problem be associated, that the ILM is not work for all indexes?

Thanks!

[RichieJarvis]

Hi Serge,

It looks like your existing (old) indices are using version 1 of the ILM Policy, which does not have a delete phase. You can see the current ILM policy being applied in your output of GET nginx-2020.01.11/_ilm/explain. The important section is this:

        "policy": "nginx_index_delete",
        "modified_date_in_millis": 1574062458159,
        "phase_definition": {
          "actions": {
            "set_priority": {
              "priority": 100
            }
          },
          "min_age": "0ms"
        },
        "version": 1
      },

As you can see, version 1 is still in-use by this index. In order to change it to use version 2, you should reapply the ILM settings to the relevant indices using the ILM API to remove the policy: https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-remove-policy.html and then readd using the Settings API, as shown here:https://www.elastic.co/guide/en/elasticsearch/reference/current/_switching_policies_for_an_index.html

Hope that helps.

Cheers,

Richie

[SergeSpinoza]

@RichieJarvis thanks!

I changed policy as you recommended, now it shows (for example):
GET nginx-2020.01.13/_ilm/explain:

{
  "indices": {
    "nginx-2020.01.13": {
      "policy": "nginx_index_delete",
      "index": "nginx-2020.01.13",
      "managed": true
    }
  }
}

In index summary I see:

that the Current action time of Index lifecycle management is 2020-01-29 12:15:50, but index is not deleted (documentation says that ilm runs every 10 min).

What else could be the problem?

[SergeSpinoza]

Maybe problem in this?

GET _ilm/status:

{
  "operation_mode": "STOPPED"
}

[SergeSpinoza]

Yes. The problem was in "operation_mode": "STOPPED".

I done POST _ilm/start and ILM began to work. Why ILM was self stoped - remains unclear.