Apply ILM Policy to existing DataStream Index

Elastic Stack Elasticsearch
1

We have an exiting datastream foo-stream and two backing indices .foo-stream-000001 and .foo-stream-000002.

Our application is writing/reading to the index foo-stream and the overall expectation is that ILM would automatically create and adjust the writable index (-00000#) as the policy thresholds are met.

The policy:

        "rollover": {
          "max_age": "1d",
          "max_size": "50gb"
        }

The problem is that the index .foo-stream-000001 is 32 days old and contains 1.1TB of data. We found that the ILM policy was not attached to this specific index, but does match the index pattern. When manually applying the ILM policy to this index, .foo-stream-000002 was created, but it contains no documents and the .foo-stream-000001 has stopped receiving documents.

I would have expected that when applying the ILM Policy to .foo-stream-000001 it would have split the data into at least 32 different indexes. What am I missing?

The explain for .foo-stream-000001

{
      "indices": {
    ".foo-stream-000001": {
      "index": ".foo-stream-000001",
      "managed": true,
      "phase_execution": {
        "policy": "foo-stream",
        "modified_date_in_millis": 1615925871750,
        "phase_definition": {
          "actions": {
            "rollover": {
              "max_age": "1d",
              "max_size": "50gb"
            },
            "set_priority": {
              "priority": 100
            }
          },
          "min_age": "0ms"
        },
        "version": 5
      },
      "action_time_millis": 1615919902092,
      "age": "2.22h",
      "step_time_millis": 1615919902092,
      "phase_time_millis": 1615918967960,
      "phase": "hot",
      "step": "complete",
      "lifecycle_date_millis": 1615919894909,
      "policy": "foo-stream",
      "action": "complete"
    }
      }
}
2

AFAIK
The policy will work from NOW. It will not calculate what should have been in retrospect.

3
  1. Is there a way to manually split the first index?
  2. Why would the first index still be receiving new documents? The .foo-stream-000002 index still has 0 documents, but the .foo-stream-000001 index continues to grow
4
  1. You can re-index
  2. You need to apply your logic in index settings, for existing indices. Elasticsearch applies templates to new indices based on an index pattern that matches the index name.
    Create or update index template API | Elasticsearch Reference [7.11] | Elastic
5

I've discovered a bit more information which might explain why this is not working as intended, and is a bit of a rabbit hole situation.

When trying to reindex from the .foo-stream-000001 index into the foo-stream to then have the ILM policy perform as expected, I get the error

no write index is defined for alias [foo-stream]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index

ILM should have automatically updated the is_write_index when creating the .foo-stream-000002 but seems like it did not. I believe the trouble there is documented in this issue: Alias incorrectly refers to data stream · Issue #67730 · elastic/elasticsearch · GitHub

Effectively the Index template contained an alias which had the same name as the DataStream. This then allowed the backing indexes to inherit the aliases, but they should not have.

So at this point it seems that both indexes are in an inconsistent state and it is unclear how to walk this back.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.