ES cloud - Reduce index size with rollover failing

Elastic Stack Elasticsearch
1

Hello,

We have an ES cloud deployment having version 7.17. The storage is being consumed fast.
One of our index has grown too big hence utilizing lot of the instance storage, I tried to setup a rollover policy on that index but that is failing because:

illegal_argument_exception: index name [myindexname] does not match pattern '^.*-\d+$'

  1. How can I rename the index without breaking the ongoing write operations to it?
  2. How can I clear the space if rollover isn't an option? I tried deleting data by query but there is no apparent change in index size although the docs count decreased.

Thanks

2

Hi @twilight

You are going to need to share a lot more details.

How did you set this up?

Please show the index template?

Please show the index?

Please show the ILM policy?

Please show the index alias?

Then perhaps we can help.
.

3

Not sure how can I export/share the index and template here.

The instance is setup on elastic cloud with many indices, this particular one is growing in size so we want to shrink or delete the older data from it.

The lifecycle policy was copied from metricbeat index lifecycle policy what has a rollover setting as well, so we have created an alias of the index which is required for rollover.

The policy has hot, warm, cold and delete phases enabled.

About size being unchanged even after deleting old data and lot of documents, I read about force merge, can you help with that? This index is active and continuously being written.

Thanks

4

Hi @twilight

Did you carefully read this

With respect to force merge

You should read this

You should only do force merge after you've rolled over and the indices is no longer being written to.

You can set force merge in the life cycle policy... But you should not really run it on an index that's still being written to.

There are APIs to get the template. Get the ILM policy. Get the aliases that is what I was hoping to see. Without that it's pretty hard to help.

You can run the APIs from the Kibana dev tool.

To help, we need to see exact, not paraphrase like I still don't know what the actual index is, what the alias is, what the ILM policy is. Without that we're just guessing

5

Does this index contain immutable data or do you perform updates?

6

We don't do updates, only write new data.

7

I understand your point.
Since policy is failing, hence no rollover is happening and unfortunately I can't do force merge as well.

I tried renaming the index using /_reindex but it created a new index with new name but same template. I am attempting to rename according to the lifecycle policy requirement so that the policy and rollover can be executed.

8

Hi @twilight We / I are trying to help but we really need more so I will try one last time...

Can you run these from Kibana Dev Tools

# if Using new Templates 
GET _template/yourtemplatename

# Or if using legacy template 
GET _index_template/yourtemplate

GET _ilm/policy/yourpolicy

GET yourindexname/_alias 

GET yourindexname/_settings

And paste in the command and the results for each of these.

Also what are you using to write the data ... logstash a beat?

The problem is probably that your processes that are writing to the index are not writing to the alias... assuming an alias even exists

9

I can't get the template, may be because we didn't use a specific one while initializing the index.

We are using it for logs, data is being pushed from different system components via POST query. Some new fields have also been added on the go.

Here is the alias info.

============================
GET ebqlogs/_alias
============================
{
  "ebqlogs" : {
    "aliases" : {
      "ebqlogs-" : { }
    }
  }
}

Here is the lifecycle policy info:

===============================
GET _ilm/policy/ebqlogs-90-days
===============================
{
  "ebqlogs-90-days" : {
    "version" : 6,
    "modified_date" : "2023-03-11T19:07:33.765Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "75d",
          "actions" : {
            "set_priority" : {
              "priority" : 50
            }
          }
        },
        "cold" : {
          "min_age" : "90d",
          "actions" : {
            "set_priority" : {
              "priority" : 0
            }
          }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "100gb",
              "max_age" : "60d"
            }
          }
        },
        "delete" : {
          "min_age" : "120d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            },
            "wait_for_snapshot" : {
              "policy" : "cloud-snapshot-policy"
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [ ],
      "data_streams" : [ ],
      "composable_templates" : [ ]
    }
  }
}

Here is the index settings:

============================
GET ebqlogs/_settings
============================
{
  "ebqlogs" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "ebqlogs",
        "creation_date" : "1594309231158",
        "number_of_replicas" : "1",
        "uuid" : "H_t-GgIwQJm53N2U-aGGnA",
        "version" : {
          "created" : "7060199"
        }
      }
    }
  }
}

I just created the alias when trying to attach a lifecycle policy, haven't used that for writing data.

Thanks

10

In order to use rollover, the index has to be set up that way on initialisation. I do not think you can convert an existing, non-rollover index to use rollover.

The easiest way to solve this might be to create a new data stream or rollover index and start writing new data to this. You can add a read alias to the template and have this cover both the new data stream/rollover index and the existing one and query through this.

To delete data from your current index you can either wait until all data is beyond your retention period and delete the whole index or you would need to delete data using delete-by-query, which will initially increase the size of the index. As no more data is being written to this index you should be able to forcemerge it down to 1 segment after the delete and reclaim the space.

1 Like
11

Can you please elaborate this? Creating a new index with same data elements / fields, can we add new fields at runtime to this new rollover-enabled index? We do add new fields sometimes into existing index as the requirement arises.
Also how do I attach one alias to both new and old index for reading older + new data and writing new data?

Thanks

12

Rollover results in a set of related time-based indices, so it is not a single index. If you want to add fields or add ti mappings you will need to do so for multiple indices.

You can assign an alias to the old index and also set this in the index template used for the rollover indoces. This will allow you to query all data. You will however perform all indexing directly to the rollover write alias, not the alias used for reading data.

13

Hello,

I was able to create new index with specific naming pattern and attached a lifecycle policy which seems to be working fine (rollover and creating new index after the specified period).

However I noticed that the new index created by the lifecycle policy doesn't have same mapping and hence there are errors popping up in Kibana (discover).
for example: date field is date/epoch_millis in first index but the subsequent index only has it as "long".

index mapping on 000001:
image

index mapping on 000002:
image

How can I make sure the subsequent auto-generated indices have same mapping as the primary/first one?

Thanks

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.