Upgrade from 5.6.8 to 6.2.2 failing

[blackey (Brad Lackey)]

I am in the process of working through a rolling update to my cluster an am not sure if its working correctly.

I fixed the issues the assistant had before starting.

After creating Certs and enabling TLS I upgraded the first node.

The first node I upgraded is complaining about authenticating my elasticsearch user.

curl -u elastic -XGET 'localhost:9200/_cat/health?pretty'
Enter host password for user 'elastic':
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "failed to authenticate user [elastic]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "failed to authenticate user [elastic]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
}

This is set to the default password.

Ideas?

[blackey (Brad Lackey)]

I am seeing this in the log.

[2018-03-07T18:11:18,110][INFO ][o.e.x.m.e.l.LocalExporter] waiting for elected master node [{pp-es-03}{QfWBnWYxTWyQFq6qgeto2Q}{q_bsqgmoTKqO5gHi6LJk4w}{10.161.5.41}{10.161.5.41:9300}{ml.max_open_jobs=10, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)

They all have x-x-pack installed.

[blackey (Brad Lackey)]

After running the ./setup-passwords interactive for x-pack and setting it to the same original password that were used I can now run API commands on the new hosts.

They show up in this list..

curl -user elastic -XGET 'localhost:9200/_cat/nodes?pretty'
10.161.6.42 11 95 1 0.14 0.09 0.03 d - pp-es-04
10.161.4.42 12 74 1 0.05 0.12 0.08 di - pp-es-05
10.161.6.41 7 99 8 0.38 0.16 0.12 md - pp-es-01
10.161.5.41 67 99 12 0.07 0.02 0.03 md * pp-es-03
10.161.4.41 8 82 20 0.12 0.11 0.23 md - pp-es-02
10.161.5.42 10 98 0 0.14 0.08 0.09 di - pp-ek-01

But don't show up in Kibana Monitoring..

[blackey (Brad Lackey)]

I solved this. I continued the upgrade and eventually they all shoed up on kibana dashboard.