7.9 -> 7.10 ECK experience

So I've recently migrated our observability cluster from Elastic 7.9 to Elastic 7.10, and noticed some interesting gotchas that are worthy of discussion. I don't think these are bugs per say but there is definitely some behaviour here that warrants further investigation, or at the very least documentation.

To give a background of our pipeline, we run Beats, which write to Logstash, which then writes to an index that setup with a custom ILM policy to do rollover (as was the way before Ingest pipelines) sat on ECK version 1.2.1

Now our ECK architecture uses a custom LB object which only selects the hot nodes from our cluster, since thats where we want all the data written to. We ensure this behaviour is seen on the Elastic side, by setting up our index templates with

"index.routing.allocation.require.data": "hot"

where data is an attribute of the hot nodes we want the data written to. This is you bog standard way of setting up ILM, or at least it was before ILM roles were introduced.

Now the reason this works, is because when ILM moves the index through its stages, it will update this field to warm and then to cold, all attributes of our nodes. With roles based ILM, this is subtly changed, by making use of the _data_tier attribute to dictate routing. This has the added benefit of allowing shards to be allocated to the tier above it should that tier run out of space (we'll touch on this again in a moment)

So then, our migration day comes, we change our node attributes and config to remove the legacy way of adding roles, add the new relevant data tiers, however for posterity we kept the existing node attributes (hot, warm, cold etc).

Cluster migrated, all goes okay, however the hot nodes are just filling up non-stop. Indexes are not being moved into the warm nodes. After finally finding a lunchtime to look at why, we discover that indexes are correctly being first written to the hot nodes (thanks to the use of the data_content role being on the hot nodes), the index the rolls over to the warm phase, and sets its _data_tier to data_warm,data_hot but then cannot actually move to those nodes, because our index template (which we did not change) had continued to put require.data: hot on each of the indexes, and the ILM policies were no longer changing them

Now I get why this happens; there exist use cases where you want to route indexes according to arbitrary node attributes, and migrating to roles based ILM should not interfere with that. That said, I doubt I'm the only person with this setup on Elastic. I'm sure there is a sufficiently interesting technical solution such as interrogating which attributes are using in ILM, and adjusting behaviour accordingly, but a simpler solution would be to just document this behaviour somewhere as a gotcha to watch out for (I tried to do this myself but I couldnt find the best place to put it other than Twitter)

The other interesting thing that I saw, was that getting the ILM policies in question, they did not include the migrate action like I would expect? The docs say that this is enabled by default and does not need explicitly adding unless you had other manual routing requirements in there as well (which of course I did at the time)

Furthermore, modifications to the ILM policies were in fact made, but only in half a capacity. Moving from hot -> warm nodes was configured to use roles, but moving from warm -> cold nodes was still configured to use attributes based routing. I'm not sure what drove that behaviour, since the cluster had nodes of all 3 roles.

I'd also highly recommend not moving to roles based ILM and upgrading to 7.10 in 1 go (ECK makes this very tempting but alas, do not).

Actions I took to fix this:

  • Removing the require.data: hot setting from all index templates
  • Manually combing through indexes which still had this setting and removing it:
PUT auditbeat-7-*/_settings
{
  "index": {
    "routing": {
      "allocation": {
        "require": {
          "data": null
        }
      }
    }
  }
}
  • Using the API (not the UI) to fix the ILM policies, including the relevant migrate{} action for all policies

I'm sure this was a bit of a corner case in terms of moving to roles based ILM, but I'd also imagine I'm not the only one who has a setup like this.

2 Likes

@jasontedor ping ^

@3266miles Thank you for taking the time for the thorough and thoughtful feedback. Feedback like this is what makes this a great community and everyone that is part of this community benefits (including Elastic).

Your points are quite valid and agree that there is no good single spot to quickly slip this documentation in somewhere. I logged https://github.com/elastic/elasticsearch/issues/65855 for us to create a migration guide from the shard filter allocation approach to the new data tier approach. I think I captured all the points here, but if not feel free to add/correct any points.

The other interesting thing that I saw, was that getting the ILM policies in question, they did not include the migrate action like I would expect?

The migrate action is an implicit action, one you will not see when you view the policy. We actually have quite a few implicit actions we inject into the workflow to ensure the correctness of certain actions. However, i do wonder if we should some how make this action a bit more obvious.

So just on that "implicit action" point, from the docs:

ILM automatically injects the migrate action in the warm and cold phases if no allocation options are specified with the allocate action.

If my ILM policies had still had allocate actions, such as to the warm node attributes, then migrate would not be implied correct? At least, thats how that reads from the docs.

Furthermore (and slightly tangentally but somewhat relevant) I'm now seeing indexes that are failing to allocate. A check of

GET _cluster/allocation/explain?pretty

Shows some interesting behaviour. First - shards are not being allocated because hot nodes do not meet the requirement for data_warm despite the index tolerating data_hot. What are the underlying conditions for the index to allow allocation to the tier above? Do all the nodes need to be at their watermark?

Secondly, I'm seeing indexes where, due to the shrink action, have _id routing requirements added to the shards, however cold nodes are being selected for this process, and not warm ones (where the shrink happens). Naturally then these shards never allocate, since it wont tolerate the _id requirements of actual warm nodes, and wont tolerate the data_warm requirement from being in the warm phase. This is either a layer 8 problem on my end, or something funky is going on.

TIA for any insight on this one

If my ILM policies had still had allocate actions, such as to the warm node attributes, then migrate would not be implied correct?

Correct. More specifically, the migrate action would not be applied if the allocate action defined any allocation rules of include, exclude, require. It is the existence of those inside the allocation that determines if migrate will be auto inject. This means that you can still set number_of_replicas in an allocate action and auto-migration would still happen.

What are the underlying conditions for the index to allow allocation to the tier above?

I think I would need to see the output of allocation explain to help here. I assume you mean the disk watermarks ? The nodes don't need to be at the disk watermark to move via ILM, but can prevent moving due to the watermark.

however cold nodes are being selected for this process

I believe that you bumped into bug: https://github.com/elastic/elasticsearch/pull/64679 where the shrink action was not taking the data tiers allocation into account when picking which node to gather the shards for the shrink action.

2 Likes

Cool. As part of the documentation then it probably helps to just drive that point home as well. I'm not sure what changes are made under the hood to ILM policies, but from what I've seen, migrate isnt being applied since they had the allocate actions in some steps (as mentioned it was split for some reason)

Glad to see I wasnt totally losing my mind then with that issue. Hope to see it in a 7.10.1 drop soon :raised_hands:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.