Automate Deletion of Docs from an Index that are older than 10 days

Elastic Stack Elasticsearch
1

Hi,

Currently, I am deleting docs using DSL queries that are older than 10 days. I am having a 1P shard on a single node with No replicas or nodes for this env.

DSL query-

POST index_name/_delete_by_query
{
 "query": {
   "range": {
     "@timestamp": {
       "lte": "now-10d"
      }
    }
  }
}

As per my understanding of ILM policy, an index is getting deleted instead of docs in an index, and a new index is created.
As I am putting ILM policy in newly created index, I need to confirm below configurations to work.
Logs are getting ingested from logstash.I guess the output filter config needs to be changed as below.

output {
      elasticsearch {
       host => <>
        ilm_rollover_alias => "index_name"
        ilm_pattern => "000001"
        ilm_policy => "new_policy"
        }
    }

Setting ILM policy
Step1-

PUT _ilm/policy/new_policy
{
    "policy": {
        "phases": {
            "hot": {
                "min_age": "0ms",
                "actions": {
                    "rollover": {
                        "max_size": "40gb"
                    },
                    "set_priority": {
                        "priority": 100
                    }
                }
            },
              "delete": {
                "min_age": "10d",
                "actions": {
                    "delete": {}
                }
            }
       
        }
     }
}
}

2 nd step creating template-

PUT _template/new_index_template
{
  "index_patterns": [
    "index_name-*"
  ],
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
      "index.lifecycle.name": "new_policy",
      "index.lifecycle.rollover_alias": "new_index"
    },
    "mappings": {<....>
   }
}

3 rd step-

PUT index-name-000001
{
  "aliases": {
    "new_index": {
      "is_write_index": true
    }
  }
}
2

ILM requires time-based indices so that complete indices can be deleted once indices exceed the retention period. This is much more efficient than using delete-by-query to periodically trim indices, which is why it is the recommended approach. If you wish to continue relying on DBQ ILM will not be able to help you and you need to continue running periodic jobs yourself.

3

Thanks @Christian_Dahlqvist ,
My index was not time series index, post reindexing it I convert it into time series index and rollover is happening as per ilm policy and able to achieve the goal.

POST /_reindex
{
  "source": {
    "index": "weather"
  },
  "dest": {
    "index": "weather-000001"
  }
}

DELETE weather

Appying Alias on timeseries index, Queries can run on alias which will be the weather.
POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "weather-000001",
        "alias": "weather",
        "is_write_index": true
      }
    }
  ]
}

Logstash output-

output {
  stdout {
    codec => rubydebug
    }
   elasticsearch {
           ilm_rollover_alias => "weather"
           ilm_pattern => "000001"
           ilm_policy => "weather"
     	   hosts => ["localhost:9200"]

One Query-

If we query (using an index pattern) for the last 3 days of data in an index that has rollover set to daily, does the query run only on the latest 3 indices or does it run on all?
I guess we need to specify the indices names in the query to limit to search to only the last 3 indices. Doesn't it happen automatically? Or does it happen automatically in the data stream but not in the index alias/ pattern?
If we have 10 days of data with daily rollover.

Sample query-

GET weather-*/_search
{
 "query": {
   "range": {
     "@timestamp": {
       "gte": "now-3d"
      }
    }
  }
}

Ref link-

Index lifecycle error - illegal_argument_exception: index.lifecycle.rollover_alias

Manage existing indices | Elasticsearch Guide [7.15] | Elastic
Data rollover in Elasticsearch

4

You can generally query all indices matching a specific pattern as Elasticsearch nowadays will rewrite the query so that indices not holding any data matching the time interval are not queried.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.