Hi Team,
We have been using EFK stack in our environment from many months( Elasticserach, kibana verison is 7.4, Xpack security also enabled, used helm method for installation) and able to access the kibana console. No issues so far. We had tried to configure new version of EFK stack ( Elasticsearch, kibana versrion is 7.9) in one of the local test cluster. Installation was sucessful, elasticsearch and kibana pods are in RUNNING state. However, we are not able to access the kibana console. We exposed kibana console using Istio service mesh ( gateway and virtualservices). I am able to access new kibana console with kubectl port-forward option, but not able to access it using URL through browser. When I check the logs of kibana console, I noticed following error messages for kibana pod.
Error message showed up when tried to access through browser.
Readiness probe for Kibana pod also failed
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 13m (x7 over 3h49m) kubelet, XXXXXX.XXXX.com Readiness probe failed: Error: Got HTTP code 503 but expected a 200
http "/kibana/app/kibana"
Kibana pod log messages.
{"type":"log","@timestamp":"2021-02-09T12:59:29Z","tags":["warning","plugins-discovery"],"pid":6,"message":"Expect plugin \"id\" in camelCase, but found: beats_management"}
{"type":"log","@timestamp":"2021-02-09T12:59:29Z","tags":["warning","plugins-discovery"],"pid":6,"message":"Expect plugin \"id\" in camelCase, but found: triggers_actions_ui"}
[BABEL] Note: The code generator has deoptimised the styling of /usr/share/kibana/x-pack/plugins/canvas/server/templates/pitch_presentation.js as it exceeds the max of 500KB.
{"type":"log","@timestamp":"2021-02-09T13:00:41Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2021-02-09T13:00:41Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"auditTrail\" is disabled."}
{"type":"log","@timestamp":"2021-02-09T13:00:42Z","tags":["warning","config","deprecation"],"pid":6,"message":"Setting [elasticsearch.username] to \"elastic\" is deprecated. You should use the \"kibana_system\" user instead."}
{"type":"log","@timestamp":"2021-02-09T13:00:42Z","tags":["warning","config","deprecation"],"pid":6,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-02-09T13:00:42Z","tags":["warning","config","deprecation"],"pid":6,"message":"Setting [monitoring.username] to \"elastic\" is deprecated. You should use the \"kibana_system\" user instead."}
{"type":"log","@timestamp":"2021-02-09T13:00:52Z","tags":["listening","info"],"pid":6,"message":"Server running at https://0.0.0.0:5601/kibana"}
{"type":"log","@timestamp":"2021-02-09T13:00:53Z","tags":["info","http","server","Kibana"],"pid":6,"message":"http server running at https://0.0.0.0:5601/kibana"}
{"type":"log","@timestamp":"2021-02-09T13:00:54Z","tags":["warning","plugins","reporting"],"pid":6,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"response","@timestamp":"2021-02-09T13:00:59Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.29.0"},"res":{"statusCode":200,"responseTime":115,"contentLength":9},"message":"GET /app/kibana 200 115ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-09T13:01:09Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.29.0"},"res":{"statusCode":200,"responseTime":11171,"contentLength":9},"message":"GET /app/kibana 200 11171ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-09T13:01:21Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.29.0"},"res":{"statusCode":200,"responseTime":2409,"contentLength":9},"message":"GET /app/kibana 200 2409ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-09T13:01:29Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.29.0"},"res":{"statusCode":200,"responseTime":99,"contentLength":9},"message":"GET /app/kibana 200 99ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-09T13:01:39Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.29.0"},"res":{"statusCode":200,"responseTime":79,"contentLength":9},"message":"GET /app/kibana 200 79ms - 9.0B"}
We defined "elastic" user name credentials in values.yaml file under secrets section. Is anything changed in new version of elasticsearch and kibana? What is the defautl password for kibana_system user?
Thanks,
Kasim -/-