Inconsistent node role display for the same configuration

Elasticsearch version: 7.16.2

In our case, all data nodes in elasticsearch.yml file are configured as

node.roles: [data_hot, data_content, ]

But running cat node in dev tool we see inconsistent node role display, some of the data nodes are marked as d, the rest is marked as hs, for example:

ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.19.0.172           38          99   5    0.74    0.60     0.60 d         -      172.19.0.214
172.19.0.166           27          99  10    0.54    0.83     0.94 hs        -      172.19.0.166
....
....
....

Does anyone know the reason for this, thanks!

Did you upgrade from an earlier version recently? If so, try another rolling restart.

Hi @DavidTurner

Yes, we upgraded from version 7.6.x. Thanks for your suggestion, will have a try (tried, it works!).

Do you know the possible reason why this happened? Kind of curious.

I think there's a small bug that affects upgrades from very old versions in this way, depending on the order in which nodes were upgraded.