Using ILM for Timeseries Data with a Static Write Index

[Karl_Falconer]

We are using a Ruby wrapper library called Searchkick which provides great functionality for searching within our application environment. However it is not great with managing Timeseries data.

Our objective is to be able to use ILM and a Hot/Cold architecture to better manage resources. Currently the library will write data into a single index and it will just grow forever. We have a temporary solution of running a _delete_by_query to purge records within the given index older than 30 days.

The library code executes searches against a single named index: foo_production_<timestamp>. The only time this index name would change is if for some reason we needed to reindex all of the data from source.(this does not happen often)

My question is, is there a way to implement a rollover strategy without making application/library code changes?

Are we able to implement an ILM policy such that the original index name always remains writeable, and we can maintain the ability to search across the newly created index. In other words can we create an alias with the same name as an index? Does it make sense to convert to a datastream? How would that impact searching?

If we can not get around making some application level change, is there documentation on how to implement the client side logic of index creation to support ILM on the server?

[rugenl (Len Rugen)]

I think you could create an alias to a new foo_production_<timestamp> and initialize the first index foo_production_<timestamp>.000001, which would initially be is_write_index: true.

To an application, an alias is an index, it can't tell. The alias points to all foo_production_<timestamp>.00xxxx, so queries work. Rollover moves the is_write_index to each new index as it's created.

I'm not sure if you would have to reindex, or if you could add the new alias to the old index.