Create ILM on the timestamp field

[Debasis_Mallick]

Hi Team,

I want to create a ILM on the timestamp field (which received as field in the log file itself as epoch time). So I want create the ILM on basis of that field. Could you please let me know how we can achieve the same. Below is the snippet from index creation where timestamp is a field part of the index.

PUT /elastic
{
"settings": 
{
 "refresh_interval": "30s",
  "number_of_shards": "6",
  "number_of_replicas": "1"
   },
  "mappings":{
    "properties":{
      "sequence":{
      "type":"keyword"
      },
    "timestamp":{
	  "type": "date",
      "format": "epoch_millis"

Thanks,
Debasis

[leandrojmp (Leandro Pereira)]

@Debasis_Mallick

I want to create a ILM on the timestamp field (which received as field in the log file itself as epoch time).

Can you provide more context on what you want to do? It is not clear what you mean by create an ILM on the timestamp field.

[Debasis_Mallick]

Hi @leandrojmp ,

The log files we are receiving from the app team contain a timestamp (epoch timestamp) as a field within the logfile itself.
Besides the timestamp, there are other fields that we have created INDEX.
Following the initial data ingest to the INDEX, its size expands, necessitating the management of the INDEX.
To effectively handle this, we recommend either splitting the index or implementing index rollover using an ILM policy. So I have been asked to create a new Index (by applying ILM policy) based on the timestamp field found in the log file received from the application team.

Could you please advise how we can achieve the same.

Thanks,
Debasis

[Debasis_Mallick]

@leandrojmp Did you get any chance to look into the above request.

Thanks,
Debasis

[Christian_Dahlqvist]

@Debasis_Mallick

To effectively handle this, we recommend either splitting the index or implementing index rollover using an ILM policy. So I have been asked to create a new Index (by applying ILM policy) based on the timestamp field found in the log file received from the application team.

ILM works on complete indices and not data within the indices. It does not support creation of new indices through reindexing, if that is what you are looking for.

The best way to use ILM is to use time-based indices, e.g. data stream. This periodically creates a new index, e.g. when the size of the last index grown beyond a specified size or a time period has passed, and all new indexing goes into this last index. Complete indices are then deleted by ILM once the age (based on rollover timestamp) of the index exceed the configured retention period. This approach generally assume your data is immutable, so if that is not the case you may need to do something different.