Unable to login to kibana

Hi Team,

I have installed es cluster v 7.16.2.

kibana is installed on two servers. If kibana service is running on both servers, then i am not able to login to kibana.
If its stopped on one of the server, then able to login to kibana.

The request to kibana server is going through aws alb.

Below are the details.

kibana config -

[root ~]# cat /etc/kibana/kibana.yml

server.port: 5601
server.host: "0.0.0.0"
server.name: kibana_1
path.data: /new/lib/kibana

elasticsearch.requestTimeout: 30000

elasticsearch.hosts: ['http://es_1:9200', 'http://es_2:9200', 'http://es_3:9200']
elasticsearch.username: "elastic"
elasticsearch.password: "xxxx"
[root ~]#

in dev tools, it is showing 401 error for https://kibana.ab.com/api/licensing/info.

statusCode is 401, error is Unauthorized, message is Unauthorized.

Below are kibana logs -

below alb health check logs are appearing continuously.

{"type":"response","@timestamp":"2021-12-13T11:18:40+00:00","tags":["api"],"pid":7789,"method":"get","statusCode":401,"req":{"url":"/api/status","method":"get","headers":{"host":"kibana2_ip:5601","connection":"close","user-agent":"ELB-HealthChecker/2.0","accept-encoding":"gzip, compressed"},"remoteAddress":"load_balancer_ip_1","userAgent":"ELB-HealthChecker/2.0"},"res":{"statusCode":401,"responseTime":26,"contentLength":66},"message":"GET /api/status 401 26ms - 66.0B"}

In AWS, in the target group, i have mentioned /api/status endpoint for health check and status code as 200,401. I have not mentioned elastic username, password there.

kibana logs from kibana1 server,

{"type":"log","@timestamp":"2021-12-13T16:18:08+00:00","tags":["info","plugins","security","routes"],"pid":15298,"message":"Logging in with provider \"basic\" (basic)"}
{"type":"response","@timestamp":"2021-12-13T16:18:08+00:00","tags":[],"pid":15298,"method":"post","statusCode":200,"req":{"url":"/internal/security/login","method":"post","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155bc-0e526e2216a26a5a1c4cb98f;Root=1-627155bc-55f6d776117ed5dc0cd14976","content-length":"181","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Safari/537.36","kbn-version":"7.16.2","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://kibana.ab.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome"},"res":{"statusCode":200,"responseTime":874,"contentLength":24},"message":"POST /internal/security/login 200 874ms - 24.0B"}

{"type":"response","@timestamp":"2021-12-13T16:18:09+00:00","tags":[],"pid":15298,"method":"get","statusCode":200,"req":{"url":"/app/home","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155bd-01dc59d87cc6c2157bf08ac9;Root=1-627155bd-3e9d32486fef51d631ffa3f3","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"same-origin","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome"},"res":{"statusCode":200,"responseTime":53,"contentLength":144988},"message":"GET /app/home 200 53ms - 141.6KB"}
{"type":"response","@timestamp":"2021-12-13T16:18:09+00:00","tags":[],"pid":15298,"method":"get","statusCode":304,"req":{"url":"/ui/legacy_light_theme.css","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-889122ce-46baa6391381b7cf25c0d832;Root=1-889122ce-2b3b46581990ec5a3d90b1c5","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"text/css,*/*;q=0.1","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"style","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","if-none-match":"\"4345f9a3b9a243edd0b81b913ba1901bce549a2b-gzip\"","if-modified-since":"Sat, 18 Dec 2021 20:13:25 GMT"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":304,"responseTime":2,"contentLength":206},"message":"GET /ui/legacy_light_theme.css 304 2ms - 206.0B"}
{"type":"response","@timestamp":"2021-12-13T16:18:09+00:00","tags":["api"],"pid":15298,"method":"get","statusCode":304,"req":{"url":"/bootstrap.js","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-889122ce-44763547711a19f17b52193d;Root=1-889122ce-664a3ef167df56420cf19e20,"sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","if-none-match":"\"cd9a30ea90e53c56bfebfb936f5b10b9a771b4c0-gzip\""},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":304,"responseTime":24,"contentLength":18652},"message":"GET /bootstrap.js 304 24ms - 18.2KB"}

{"type":"response","@timestamp":"2021-12-13T16:18:16+00:00","tags":[],"pid":15298,"method":"post","statusCode":200,"req":{"url":"/api/core/capabilities","method":"post","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155c4-109e0ddb0ce8dada09544630;Root=1-627155c4-1a3cdb2030031f9b7e3f4039","content-length":"589","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Safari/537.36","kbn-version":"7.16.2","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://kibana.ab.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":123,"contentLength":4774},"message":"POST /api/core/capabilities 200 123ms - 4.7KB"}
{"type":"response","@timestamp":"2021-12-13T16:18:16+00:00","tags":[],"pid":15298,"method":"get","statusCode":200,"req":{"url":"/46307/bundles/plugin/security/8.0.0/security.chunk.6.js","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155c4-08b1ae68100b0f8900780ff4;Root=1-627155c4-19c1df697bae1b963f73773e","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 
Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":4,"contentLength":882},"message":"GET /46307/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.chunk.74.js 200 4ms - 882.0B"}

{"type":"response","@timestamp":"2021-12-13T16:18:16+00:00","tags":[],"pid":15298,"method":"get","statusCode":200,"req":{"url":"/internal/security/security_checkup/state","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155c5-3fd121b25bf3ccda3a813047;Root=1-627155c5-53d7e2310bdacaac071a822e","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.16.2","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":23,"contentLength":22},"message":"GET /internal/security/security_checkup/state 200 23ms - 22.0B"}
{"type":"response","@timestamp":"2021-12-13T16:18:16+00:00","tags":[],"pid":15298,"method":"get","statusCode":200,"req":{"url":"/api/banners/info","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155c5-085270ae73f87ed03919780a;Root=1-627155c5-5dc5941431fdc37c595f171f","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 e"},"res":{"statusCode":200,"responseTime":43,"contentLength":118},"message":"GET /api/banners/info 200 43ms - 118.0B"}
{"type":"response","@timestamp":"2021-12-13T16:18:16+00:00","tags":[],"pid":15298,"method":"get","statusCode":200,"req":{"url":"/api/licensing/info","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155c5-741e9273549b785e3b3ebfb7;Root=1-627155c5-54f271203579cc76171a950b","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.16.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":17},"message":"GET /api/licensing/info 200 17ms"}

{"type":"response","@timestamp":"2021-12-13T16:18:17+00:00","tags":[],"pid":15298,"method":"get","statusCode":200,"req":{"url":"/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155c5-2abc856f5e691c163090c3bf;Root=1-627155c5-61f7c4882975560765791b66","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"same-origin","sec-fetch-mode":"navigate","sec-fetch-dest":"document","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, 
{"type":"response","@timestamp":"2021-12-13T16:18:17+00:00","tags":[],"pid":15298,"method":"get","statusCode":200,"req":{"url":"/internal/security/session","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, Safari/537.36","kbn-version":"7.16.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":247,"contentLength":85},"message":"GET /internal/security/session 200 247ms - 85.0B"}
Safari/537.36","kbn-version":"7.16.2","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":46,"contentLength":15},"message":"GET /api/index_patterns/has_user_index_pattern 200 46ms - 15.0B"}
{"type":"response","@timestamp":"2021-12-13T16:18:17+00:00","tags":[],"pid":15298,"method":"get","statusCode":200,"req":{"url":"/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155c6-51fd91a855854b21567160a8;Root=1-627155c6-031e2789089af1e365f87548","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"same-origin","sec-fetch-mode":"navigate","sec-fetch-dest":"document","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":18,"contentLength":144871},"message":"GET /logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F 200 18ms - 141.5KB"}
{"type":"response","@timestamp":"2021-12-13T16:18:18+00:00","tags":[],"pid":15298,"method":"get","statusCode":200,"req":{"url":"/46307/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.chunk.210.js","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155c6-6de03e98107ee71a46f56366;Root=1-627155c6-3ae74f33260d1cdf61a45a34","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":4,"contentLength":698},"message":"GET /46307/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.chunk.210.js 200 4ms - 698.0B"}
{Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":5,"contentLength":620},"message":"GET /46307/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.chunk.99.js 200 5ms - 620.0B"}
Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":5,"contentLength":727},"message":"GET /46307/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.chunk.166.js 200 5ms - 727.0B"}
 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":5,"contentLength":815},"message":"GET /46307/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.chunk.154.js 200 5ms - 815.0B"}
Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":5,"contentLength":682},"message":"GET /46307/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.chunk.43.js 200 5ms - 682.0B"}
Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":4,"contentLength":706},"message":"GET /46307/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.chunk.331.js 200 4ms - 706.0B"}
 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://kibana.ab.com/app/home","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/app/home"},"res":{"statusCode":200,"responseTime":4,"contentLength":763},"message":"GET /46307/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.chunk.28.js 200 4ms - 763.0B"} Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"text/css,*/*;q=0.1","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"style","referer":"https://kibana.ab.com/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","if-none-match":"\"4345f9a3b9a243edd0b81b913ba1901bce549a2b-gzip\"","if-modified-since":"Sat, 18 Dec 2021 20:13:25 GMT"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F"},"res":{"statusCode":304,"responseTime":3,"contentLength":206},"message":"GET /ui/legacy_light_theme.css 304 3ms - 206.0B"}

{"type":"response","@timestamp":"2021-12-13T16:18:23+00:00","tags":[],"pid":15298,"method":"post","statusCode":200,"req":{"url":"/api/core/capabilities?useDefaultCapabilities=true","method":"post","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155cb-18cb4b744d702ec837a88b4d;Root=1-627155cb-5086d1560c67713d2059ddfb","content-length":"589","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.16.2","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://kibana.ab.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.ab.com/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F"},"res":{"statusCode":200,"responseTime":86,"contentLength":4774},"message":"POST /api/core/capabilities?useDefaultCapabilities=true 200 86ms - 4.7KB"}
{"type":"response","@timestamp":"2021-12-13T16:18:23+00:00","tags":[],"pid":15298,"method":"get","statusCode":200,"req":{"url":"/api/banners/info","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155cb-262be0b56889ece451d20a6d;Root=1-627155cb-6fd7e5bc7d09de0465adf9b5","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.16.2","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.ab.com/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F"},"res":{"statusCode":200,"responseTime":27,"contentLength":118},"message":"GET /api/banners/info 200 27ms - 118.0B"}
{"type":"response","@timestamp":"2021-12-13T16:18:23+00:00","tags":["security:canRedirect","security:authFlow"],"pid":15298,"method":"get","statusCode":302,"req":{"url":"/api/security/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155cb-6958dbe41deafafe72e539ac;Root=1-627155cb-30862b83630119a728ced9d8","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"same-origin","sec-fetch-mode":"navigate","sec-fetch-dest":"document","referer":"https://kibana.ab.com/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F"},"res":{"statusCode":302,"responseTime":677},"message":"GET /api/security/logout?msg=AUTHENTICATION_ERROR&next=%2Fapp%2Fhome%23%2F 302 677ms"}
{"type":"response","@timestamp":"2021-12-13T16:18:24+00:00","tags":[],"pid":15298,"method":"get","statusCode":304,"req":{"url":"/node_modules/@kbn/ui-framework/dist/kui_light.css","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155cc-604cae723303b6487c525d8e;Root=1-627155cc-2701c4116a565ed542e3e568","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"text/css,*/*;q=0.1","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"style","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome%23%2F&msg=AUTHENTICATION_ERROR","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","if-none-match":"\"47af9b8f6e4640c400d5d0b64bd8c8095c9bbcf5-gzip\"","if-modified-since":"Sat, 18 Dec 2021 20:13:23 GMT"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome%23%2F&msg=AUTHENTICATION_ERROR"},"res":{"statusCode":304,"responseTime":2,"contentLength":205},"message":"GET /node_modules/@kbn/ui-framework/dist/kui_light.css 304 2ms - 205.0B"}
{"type":"response","@timestamp":"2021-12-13T16:18:26+00:00","tags":[],"pid":15298,"method":"get","statusCode":304,"req":{"url":"/translations/en.json","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155cf-378203841e1c40c3235592df;Root=1-627155cf-5bfdc57c1fd4e50501eb201a","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome%23%2F&msg=AUTHENTICATION_ERROR","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","if-none-match":"\"37992637719f97813c3068cfbf877b2d3bb43b97\""},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome%23%2F&msg=AUTHENTICATION_ERROR"},"res":{"statusCode":304,"responseTime":2,"contentLength":29},"message":"GET /translations/en.json 304 2ms - 29.0B"}

{"type":"response","@timestamp":"2021-12-13T16:18:28+00:00","tags":[],"pid":15298,"method":"get","statusCode":401,"req":{"url":"/api/licensing/info","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155d0-7d5a3903447b8c0f78750228;Root=1-627155d0-65a4a04579019df95c2e8923","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.16.2","kbn-system-request":"true","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome%23%2F&msg=AUTHENTICATION_ERROR","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome%23%2F&msg=AUTHENTICATION_ERROR"},"res":{"statusCode":401,"responseTime":24,"contentLength":66},"message":"GET /api/licensing/info 401 24ms - 66.0B"}
{"type":"response","@timestamp":"2021-12-13T16:18:28+00:00","tags":[],"pid":15298,"method":"get","statusCode":200,"req":{"url":"/internal/security/login_state","method":"get","headers":{"x-forwarded-for":"my_public_ip, ip1, ip2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kibana.ab.com","x-amzn-trace-id":"Self=1-627155d0-14f11bbf491e6c1a78599f90;Root=1-627155d0-717659a95eb9ed095615e98e","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","kbn-version":"7.16.2","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome%23%2F&msg=AUTHENTICATION_ERROR","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"load_balancer_ip_1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36","referer":"https://kibana.ab.com/login?next=%2Fapp%2Fhome%23%2F&msg=AUTHENTICATION_ERROR"},"res":{"statusCode":200,"responseTime":3,"contentLength":186},"message":"GET /internal/security/login_state 200 3ms - 186.0B"}

Can someone please help on why unable to login when service is running on both servers.

Thanks,

I don't know much about the aws load balancing service, but it seems like that's the core issue here.

Does it work if you provide the auth details?

@warkolm,

Not. I am not seeing the option anywhere to mention username, password, hence added 401 to pass the health check.

Thanks,