Looking for the best way to manage a somewhat "hot" and "warm" environment for indices

Elastic Stack Elasticsearch
1

Hello,

First, thank you for your patience while I attempt to illustrate my issue. I hope I'm being coherent enough.

I am using Apache NiFi to route data into Elasticsearch. Data being ingested into Elasticsearch has indices dynamically created based on hostname + log_type. For example: 'host1_syslog`. I'm only ingesting the current running message log in these instances. There are cases where we need to backfill message data. Doing so creates a monolithic index with historical data added.

What I'd like to do is separate the indices by current message data and historical data. Let say current message data is most recent 30 days. I could have the current message data going to host1_syslog_current and host1_syslog_old These could be aliased together host1_syslog but I'm not sure how to handle the index lifecycle policy management for that. The ilm policy just controls the rollover per index. Is there a way to create an ilm policy that would have host1_syslog_current only contain 30 days worth of data and will rollover data/append older than 30 days to host1_syslog_old

Am I explaining this well enough?

Thanks

2

ILM isn't designed to do that, as the stack expects that you have a single alias to interact with the underlying indices with.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.