As with many solutions that require retention policy for data stored in elasticsearch, we’ve implemented the commonly used pattern of having a single index per day. With this pattern removing the old data would be as simple as deleting an index. Since we are using the aws elasticsearch service, the easiest way to implement the above solution was to define an ISM policy to perform index rollover every day and index deletion after 30 days. So far so good…
Now, the problem started when we realized that the index aliases we used for searching the data now had to keep track of 30 indexes instead of just one. Using the aliases api, it is possible to make an alias point to multiple indexes e.g.: { "add" : { "index" : "daily_index*", "alias" : "30days" } } but, unfortunately, the above will only work for the indexes existing in elasticsearch at the time of the api call. New indexes added by the ISM as part of the rollover action will not be added to the ‘30days’ alias. As a side note, we need to use aliases as a safety measure in multi-tenant scenario where each alias is a filtered one with its unique tenant id e.g.: "filter" : { "match" : { "tenant_id": 123} }. This ensures that our searches return results pertained to a particular tenant. So we need to figure out a way to update our aliases manually following new index creation each day. This turns out to be not an easy task taking into account the fact that index rollover is an atomic operation performed by the ISM behind the scenes (basically speaking, we don’t have any control of when this rollover happens). We could use the elasticsearch curator instead of the ISM where it would be possible to intervene into the process updating all our aliases right after the rollover call but, then again, this update would not be atomic with some aliases pointing at the newly created index while others still pending to be updated. My question is whether an alternative to using filtered alias exists in elasticsearch apis. Ideally, if there was a way to dynamically add a term filter to every request to elasticsearch (without the need to modify the query body), that would be it. Will appreciate any help.
Are you not including the aliases in the index template so it is automatically added to all new indices?
Unfortunately, I cannot use index template. My aliases are created dynamically with each new user signup.
I do not see why you could not create an ad Lisa as Nd update an index template when you onboard a new user. Having an ad Lucas per user will scale up to a certain level, but not infinitely.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.