KIbana failover to a healthy ES node

zaeemmasood · May 16, 2023, 1:29pm

Hi All,

We are facing an issue where Kibana does not failover to a healthy Elasticsearch node in case a node goes down. Login into Kibana console stops working unless the bad node is brought up.

KIbana config is as follows:

server.port: 5601
server.host: "hostA"
server.name: "hostA"
server.basePath: "/elktool"
server.rewriteBasePath: true
# The URLs of the Elasticsearch instances to use for all your queries.

elasticsearch.username: "kibana"
elasticsearch.password: "xxxxxxxxxxxxxxxxx"

elasticsearch.hosts:
  - http://hostA:43045
  - http://hostB:43045
  - http://hostC:43045
  - http://hostD:43045
  - http://hostE:43045
  - http://hostF:43045

# Specifies the path where Kibana creates the process ID file.
pid.file: /opt/total/kibana/kibana.pid

xpack.security.session.idleTimeout: "8h"
xpack.security.loginAssistanceMessage: "Welcome to Kibana!"
xpack.security.sessionTimeout: 3600000
xpack.security.encryptionKey: "something_at_least_32_characters"
xpack.reporting.enabled: true
xpack.reporting.encryptionKey: "something__not_at_least_32_characters"

logging.rotate:
  enabled: true
  everyBytes: 10485760
  keepFiles: 10

Elasticsearch config is as follows:

cluster.name: "cluster-prod"
node.name: "hostA"
node.master: true
node.data: true

network.host: "hostA"
http.port: 43045

cluster.initial_master_nodes: ["hostA","hostB","hostC"]
discovery.zen.ping.unicast.hosts: ["hostA","hostB","hostC","hostD","hostE","hostF"]
discovery.zen.minimum_master_nodes: 2

bootstrap.memory_lock: true
bootstrap.system_call_filter: false

logger.org.elasticsearch.cluster.coordination.ClusterBootstrapService: DEBUG
logger.org.elasticsearch.discovery: DEBUG

#reindex.remote.whitelist: 
path.data: /opt/total/elasticsearch/data
path.logs: /opt/total/elasticsearch/logs/
path.repo: ["/opt/total/elasticsearch/.snapshots"]

http.cors.enabled: true
http.cors.allow-origin: "*"

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

Question is if there is a tag needed to be defined in Kibana config to enable it to failover to a healthy ES node in case an ES node goes bad?

warkolm · May 17, 2023, 12:03am

Can you share your Kibana log?

zaeemmasood · May 17, 2023, 3:56pm

Hi @warkolm

Kibana log is ~700M in size.

Anything particular you want me to check please?

Is there a tag that could enforce failover?

Thanks!

warkolm · May 17, 2023, 11:56pm

Can you grab the last 50 or so lines for us to look at? We'd be after things like node disconnects-connects.

zaeemmasood · May 18, 2023, 6:35pm

Hi @warkolm

Please see few lines with the "disconnect" error showing below:

{"type":"response","@timestamp":"2023-05-15T01:52:02-04:00","tags":[],"pid":21574,"method":"get","statusCode":200,"req":{"url":"/internal/security/login_state","method":"get","headers":{"xxxxxxxxxxxxxxxx-requestid":"xxxxxxxx","xxxxxxxxxxxxxxxx-auth-level":"3","xxxxxxxxxxxxxxxx-auth-age":"3","xxxxxxxxxxxxxxxx-auth-type":"DESKTOP","xxxxxxxxxxxxxxxx-idle-time":"0","xxxxxxxxxxxxxxxx-ver":"1.0","xxxxxxxxxxxxxxxx-environment":"prod","x-portal-env":"internal","xxxxxxxxxxxxxxxx-netzone":"internal","xxxxxxxxxxxxxxxx-region":"NAM","xxxxxxxxxxxxxxxx-csrftoken":"PCVJ3WSIUUKUR6J3I9YXMJM7JCM93FG5556226104@1","xxxxxxxxxxxxxxxx-ua-track":"AmNFJmrSt4cLwi3tEfWUw1Z3bCFG","xxxxxxxxxxxxxxxx-csid":"AmNFJmrSt4cLwi3tEfWUw1Z3bCFG","xxxxxxxxxxxxxxxx-platform":"Desktop","xxxxxxxxxxxxxxxx-enablecds":"Y","xxxxxxxxxxxxxxxx-enable-glassbox":"Y","xxxxxxxxxxxxxxxx-loginid":"xxxxxxxx","xxxxxxxxxxxxxxxx-sigkey":"xxxxxxxxxxxx","xxxxxxxxxxxxxxxx-authtime":"1684129919539","xxxxxxxxxxxxxxxx-sig":"MCwCFF7nKv32Uz1ity2tdYnwhHwPeWFRAhQ9dtrItU3bQOLeQwcvBtz0AjaAbw==","true-client-ip":"xxxxxxxxxxxx","xxxxxxxxxxxxxxxx-ip":"xxxxxxxxxxxx,xxxxxxxxxxxx,xxxxxxxxxxxxxxxx","xxxxxxxxxxxxxxxx-enable-institutional":"false","xxxxxxxxxxxxxxxx-profile":"Institutional New","xxxxxxxxxxxxxxxx-menu-type":"Modern Menu","accept":"*/*","x-portalname":"xxxxxxxx","kbn-version":"7.13.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.112.0.0 Safari/537.36","referer":"https://xxxxxxxxxxxx/xxxxxxxxxxxxelk/s/xxxxxxxxxxxx/login?next=%2Fxxxxxxxxxxxxelk%2Fs%2Fxxxxxxxxxxxx%2Fapp%2Fdiscover","x-forwarded-proto":"https","x-forwarded-host":"xxxxxxxxxxxx","sec-fetch-site":"same-origin","sec-fetch-dest":"empty","host":"xxxxxxxxxxxx","accept-encoding":"gzip, deflate, br","sec-fetch-mode":"cors","sec-ch-ua":"\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","x-forwarded-for":"xxxxxxxxxxxx","accept-language":"en-US,en;q=0.9","x-forwarded-server":"xxxxxxxxxxxx","content-type":"application/json","content-length":"0"},"remoteAddress":"xxxxxxxx","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.112.0.0 Safari/537.36","referer":"https://xxxxxxxxxxxx/xxxxxxxxxxxxelk/s/xxxxxxxxxxxx/login?next=%2Fxxxxxxxxxxxxelk%2Fs%2Fxxxxxxxxxxxx%2Fapp%2Fdiscover"},"res":{"statusCode":200,"responseTime":4,"contentLength":186},"message":"GET /internal/security/login_state 200 4ms - 186.0B"}
{"type":"response","@timestamp":"2023-05-15T01:52:02-04:00","tags":[],"pid":21574,"method":"get","statusCode":200,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-SemiBold.woff2","method":"get","headers":{"xxxxxxxxxxxxxxxx-requestid":"xxxxxxxx","xxxxxxxxxxxxxxxx-auth-level":"3","xxxxxxxxxxxxxxxx-auth-age":"3","xxxxxxxxxxxxxxxx-auth-type":"DESKTOP","xxxxxxxxxxxxxxxx-idle-time":"0","xxxxxxxxxxxxxxxx-ver":"1.0","xxxxxxxxxxxxxxxx-environment":"prod","x-portal-env":"internal","xxxxxxxxxxxxxxxx-netzone":"internal","xxxxxxxxxxxxxxxx-region":"NAM","xxxxxxxxxxxxxxxx-csrftoken":"PCVJ3WSIUUKUR6J3I9YXMJM7JCM93FG5556226104@1","xxxxxxxxxxxxxxxx-ua-track":"AmNFJmrSt4cLwi3tEfWUw1Z3bCFG","xxxxxxxxxxxxxxxx-csid":"AmNFJmrSt4cLwi3tEfWUw1Z3bCFG","xxxxxxxxxxxxxxxx-platform":"Desktop","xxxxxxxxxxxxxxxx-enablecds":"Y","xxxxxxxxxxxxxxxx-enable-glassbox":"Y","xxxxxxxxxxxxxxxx-loginid":"xxxxxxxx","xxxxxxxxxxxxxxxx-sigkey":"xxxxxxxxxxxx","xxxxxxxxxxxxxxxx-authtime":"1684129919539","xxxxxxxxxxxxxxxx-sig":"MCwCFF7nKv32Uz1ity2tdYnwhHwPeWFRAhQ9dtrItU3bQOLeQwcvBtz0AjaAbw==","true-client-ip":"xxxxxxxxxxxx","xxxxxxxxxxxxxxxx-ip":"xxxxxxxxxxxx,xxxxxxxxxxxx,xxxxxxxxxxxxxxxx","xxxxxxxxxxxxxxxx-enable-institutional":"false","xxxxxxxxxxxxxxxx-profile":"Institutional New","xxxxxxxxxxxxxxxx-menu-type":"Modern Menu","origin":"https://xxxxxxxxxxxx","accept":"*/*","x-portalname":"xxxxxxxx","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.112.0.0 Safari/537.36","referer":"https://xxxxxxxxxxxx/xxxxxxxxxxxxelk/s/xxxxxxxxxxxx/login?next=%2Fxxxxxxxxxxxxelk%2Fs%2Fxxxxxxxxxxxx%2Fapp%2Fdiscover","x-forwarded-proto":"https","x-forwarded-host":"xxxxxxxxxxxx","sec-fetch-site":"same-origin","sec-fetch-dest":"font","host":"xxxxxxxxxxxx","accept-encoding":"gzip, deflate, br","sec-fetch-mode":"cors","sec-ch-ua":"\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","x-forwarded-for":"xxxxxxxxxxxx","accept-language":"en-US,en;q=0.9","x-forwarded-server":"xxxxxxxxxxxx"},"remoteAddress":"xxxxxxxx","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.112.0.0 Safari/537.36","referer":"https://xxxxxxxxxxxx/xxxxxxxxxxxxelk/s/xxxxxxxxxxxx/login?next=%2Fxxxxxxxxxxxxelk%2Fs%2Fxxxxxxxxxxxx%2Fapp%2Fdiscover"},"res":{"statusCode":200,"responseTime":55,"contentLength":94752},"message":"GET /ui/fonts/inter_ui/Inter-UI-SemiBold.woff2 200 55ms - 92.5KB"}
{"type":"response","@timestamp":"2023-05-15T01:52:02-04:00","tags":[],"pid":21574,"method":"get","statusCode":200,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-Light-BETA.woff2","method":"get","headers":{"xxxxxxxxxxxxxxxx-requestid":"xxxxxxxx","xxxxxxxxxxxxxxxx-auth-level":"3","xxxxxxxxxxxxxxxx-auth-age":"3","xxxxxxxxxxxxxxxx-auth-type":"DESKTOP","xxxxxxxxxxxxxxxx-idle-time":"0","xxxxxxxxxxxxxxxx-ver":"1.0","xxxxxxxxxxxxxxxx-environment":"prod","x-portal-env":"internal","xxxxxxxxxxxxxxxx-netzone":"internal","xxxxxxxxxxxxxxxx-region":"NAM","xxxxxxxxxxxxxxxx-csrftoken":"PCVJ3WSIUUKUR6J3I9YXMJM7JCM93FG5556226104@1","xxxxxxxxxxxxxxxx-ua-track":"AmNFJmrSt4cLwi3tEfWUw1Z3bCFG","xxxxxxxxxxxxxxxx-csid":"AmNFJmrSt4cLwi3tEfWUw1Z3bCFG","xxxxxxxxxxxxxxxx-platform":"Desktop","xxxxxxxxxxxxxxxx-enablecds":"Y","xxxxxxxxxxxxxxxx-enable-glassbox":"Y","xxxxxxxxxxxxxxxx-loginid":"xxxxxxxx","xxxxxxxxxxxxxxxx-sigkey":"xxxxxxxxxxxx","xxxxxxxxxxxxxxxx-authtime":"1684129919539","xxxxxxxxxxxxxxxx-sig":"MCwCFF7nKv32Uz1ity2tdYnwhHwPeWFRAhQ9dtrItU3bQOLeQwcvBtz0AjaAbw==","true-client-ip":"xxxxxxxxxxxx","xxxxxxxxxxxxxxxx-ip":"xxxxxxxxxxxx,xxxxxxxxxxxx,xxxxxxxxxxxxxxxx","xxxxxxxxxxxxxxxx-enable-institutional":"false","xxxxxxxxxxxxxxxx-profile":"Institutional New","xxxxxxxxxxxxxxxx-menu-type":"Modern Menu","origin":"https://xxxxxxxxxxxx","accept":"*/*","x-portalname":"xxxxxxxx","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.112.0.0 Safari/537.36","referer":"https://xxxxxxxxxxxx/xxxxxxxxxxxxelk/s/xxxxxxxxxxxx/login?next=%2Fxxxxxxxxxxxxelk%2Fs%2Fxxxxxxxxxxxx%2Fapp%2Fdiscover","x-forwarded-proto":"https","x-forwarded-host":"xxxxxxxxxxxx","sec-fetch-site":"same-origin","sec-fetch-dest":"font","host":"xxxxxxxxxxxx","accept-encoding":"gzip, deflate, br","sec-fetch-mode":"cors","sec-ch-ua":"\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","x-forwarded-for":"xxxxxxxxxxxx","accept-language":"en-US,en;q=0.9","x-forwarded-server":"xxxxxxxxxxxx"},"remoteAddress":"xxxxxxxx","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.112.0.0 Safari/537.36","referer":"https://xxxxxxxxxxxx/xxxxxxxxxxxxelk/s/xxxxxxxxxxxx/login?next=%2Fxxxxxxxxxxxxelk%2Fs%2Fxxxxxxxxxxxx%2Fapp%2Fdiscover"},"res":{"statusCode":200,"responseTime":53,"contentLength":96700},"message":"GET /ui/fonts/inter_ui/Inter-UI-Light-BETA.woff2 200 53ms - 94.4KB"}
{"type":"log","@timestamp":"2023-05-15T01:52:03-04:00","tags":["warning","environment"],"pid":21574,"message":"Detected an unhandled Promise rejection.\nResponseError: [xxxxxxxxxxxxxxxx] [hostname] [xxx.xx.xx.123:9300][indices:data/read/get[s]] disconnected: node_disconnected_exception"}

system · June 15, 2023, 6:35pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.