Updated nodes no longer show up in Kibana monitoring

[JSieminski (John)]

I have a 8 server cluster, with each server running 4 instances of elasticsearch on it (ie.. ELK001, ELK001-a, ELK001-b, ELK001-c). I have started the rolling update process from 7.7 to 7.13.4. After updating ELK001 to 7.13.4 it is no longer showing up in Kibana monitoring. However, when i run GET _cat/nodes I see the nodes. Any assistance would be welcomed.

10.2.40.230 39 94 39 13.79 13.66 13.60 dilrt - elk003-b
IP.244 22 97 34 11.04 11.52 11.69 dilrt - elk008-a
IP.229 31 98 44 12.38 12.47 12.74 dilrt - elk002-c
IP.229 21 98 37 12.38 12.47 12.74 m - elk002
IP.242 27 96 36 11.84 12.60 12.88 dilrt - elk006-a
IP.229 23 98 51 12.38 12.47 12.74 dilrt - elk002-b
IP.243 23 94 32 10.53 10.33 10.25 dilrt - elk007-b
IP.228 4 40 1 0.32 0.36 0.53 dfilrt - elk001-a
IP.241 26 97 41 14.10 14.31 14.26 dilrt - elk005-a
IP.230 73 94 37 13.79 13.66 13.60 m - elk003
IP.230 39 94 40 13.79 13.66 13.60 dilrt - elk003-c
IP.241 32 97 41 14.10 14.31 14.26 dilrt - elk005-b
IP.242 28 96 37 11.84 12.60 12.88 dilrt - elk006-c
IP.243 31 94 32 10.53 10.33 10.25 dilrt - elk007-a
IP.228 12 40 0 0.32 0.36 0.53 m - elk001
IP.231 35 93 35 10.74 11.42 11.65 dilrt - elk004-a
IP.231 27 93 39 10.74 11.42 11.65 dilrt - elk004-b
IP.241 23 97 40 14.10 14.31 14.26 m - elk005
IP.244 41 97 35 11.04 11.52 11.69 m * elk008
IP.231 29 93 34 10.74 11.42 11.65 dilrt - elk004-c
IP.242 22 96 36 11.84 12.60 12.88 dilrt - elk006-b
IP.244 59 97 34 11.04 11.52 11.69 dilrt - elk008-c
IP.230 28 94 40 13.79 13.66 13.60 dilrt - elk003-a
IP.229 51 98 41 12.38 12.47 12.74 dilrt - elk002-a
IP.242 27 96 36 11.84 12.60 12.88 m - elk006
IP.228 4 40 1 0.32 0.36 0.53 dfilrt - elk001-b
IP.244 39 97 33 11.04 11.52 11.69 dilrt - elk008-b
IP.241 22 97 41 14.10 14.31 14.26 dilrt - elk005-c
IP.231 45 93 33 10.74 11.42 11.65 m - elk004
IP.243 37 94 33 10.53 10.33 10.25 dilrt - elk007-c
IP.243 21 94 33 10.53 10.33 10.25 m - elk007

I'm also getting the following in the logs: [2021-07-27T16:13:31,184][INFO ][o.e.x.m.e.l.LocalExporter] [elk001] waiting for elected master node [{elk008}{gLUUnIY-SdCwcMiKctQpwg}{cela3dfFSpK6Dh-qvGf6PA}{IP.244}{IP.244:9300}{m}{xpack.installed=true, transform.node=false}] to setup local exporter [default_local] (does it have x-pack installed?)

I do believe I have x-pack installed on both machines.

[rugenl (Len Rugen)]

Are you monitoring your nodes with metricbeat? Did you update metricbeat at the same time on each node?

I think the rolling update process is to update elasticsearch first on all nodes, then Kibana, then beats, that is from memory, but the key is update elasticsearch first.

Does the "waiting for elected master node" message stop after awhile or continue? I chased something that was just a timing issue at node startup and this looks familiary.

[JSieminski (John)]

This was the first node in the update process for me. I have only updated elasticsearch at this time (the game plan is to update elasticsearch on all nodes). Looks like it continues to spit out "waiting for elected master node" .

[warkolm (Mark Walkom)]

Welcome to our community!

What do the logs from the upgraded node show?