Cannot set password for new user Kibana ECK 7.4.2

Brand new cluster running on kubernetes ECK 7.4.2, open up Kibana log in as elastic user, create a new user, log out and cannot log in with wrong password.
I've tried the obvious, deleted the user, changed the password to 123456, created another user with same password. No luck.

Hi Oaaraya,

Can you let me know what roles (if any) you applied to the user?

Superuser, so I did a fresh install and logged in as elastic could not set the password for my user. I also added several roles and created a role that has every permission for Kibana. No luck.

Did you add any roles that don't exist in Elasticsearch?

I did to add all kibana permissions. Should I remove it? No other role has any kibana permissions, that seemed odd to me but I was running 7.4.0 with no issues so I just never really see the roles in detail, just assigning was always enough.

Let's just try it out without the roles and see if that is the issue. It might not be but I remember there was an issue around permissions that kept giving (invalid username and password) error even when it was for another reason.

Tried that, even created a new user named test and password 123456. Same result. The only user I can log in with is elastic.

Do you have any logs that we can look at? Or any stack trace when you try to login in?

{"type":"response","@timestamp":"2019-11-13T16:43:02Z","tags":[],"pid":12,"method":"post","statusCode":401,"req":{"url":"/api/security/v1/login","method":"post","headers":{"host":"xxxx","content-length":"45","accept":"application/json, text/plain, */*","origin":"xxx","kbn-version":"7.4.2","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36","content-type":"application/json;charset=UTF-8","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","referer":"https://xxxx/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,es;q=0.8","x-cloud-trace-context":"966f00dcd9b8a2dce514b192347402a8/6570964867542393934","via":"1.1 google","x-forwarded-for":"xxxxxx,"x-forwarded-proto":"https","connection":"Keep-Alive"},"remoteAddress":"10.150.15.201","userAgent":"10.150.15.201","referer":"https://xxx/login?next=%2F"},"res":{"statusCode":401,"responseTime":25,"contentLength":9},"message":"POST /api/security/v1/login 401 25ms - 9.0B"}

{"type":"response","@timestamp":"2019-11-13T16:43:08Z","tags":,"pid":12,"method":"get","statusCode":200,"req":{"url":"/login","method":"get","headers":{"host":"10.20.9.11:5601","user-agent":"kube-probe/1.14+","accept-encoding":"gzip","connection":"close"},"remoteAddress":"10.20.9.1","userAgent":"10.20.9.1"},"res":{"statusCode":200,"responseTime":16,"contentLength":9},"message":"GET /login 200 16ms - 9.0B"}

I've tested further and I think is an elasticsearch authentication issue
Created the user:
curl -u elastic:$PASSWORD -k -X POST "https://localhost:9200/_security/user/test1?pretty" -H 'Content-Type: application/json' -d'
{
"password" : "123456",
"roles" : [ "superuser" ],
"full_name" : "Jack Nicholson",
"email" : "jacknich@example.com",
"metadata" : {
"intelligence" : 7
}
}
'
Tested the user:
curl -u "test1:123456" -k "https://localhost:9200/_cluster/health"

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [test1] for REST request [/_cluster/health]","header":{"WWW-Authenticate":["Bearer realm="security"","ApiKey","Basic realm="security" charset="UTF-8""]}}],"type":"security_exception","reason":"unable to authenticate user [test1] for REST request [/_cluster/health]","header":{"WWW-Authenticate":["Bearer realm="security"","ApiKey","Basic realm="security" charset="UTF-8""]}},"status":401} %

Found the solution here:

There is a bug and the realm is not set for ECK