Stack monitoring is broken in 7.9.1

Elastic Stack Kibana
#1

Stack monitoring displays a blank white screen. No errors in Kibana logs. It worked fine in 7.8.0, upgrade to 7.9.1 broke it. I have verified that the monitoring data is flowing to the right indices. Below is my Kibana.yml.

        console.enabled: false
        csp.strict: true
        elasticsearch.hosts: '["http://es-host1:9200","http://es-host2:9200","http://es-host3:9200","http://es-host4:9200","http://es-host5:9200"]'
        elasticsearch.pingTimeout: '2500'
        elasticsearch.requestTimeout: '300000'
        logging.dest: "/var/log/kibana/kibana.stdout"
        logging.verbose: false
        monitoring.enabled: true
        monitoring.ui.elasticsearch.hosts: '["http://es-monitor:9200"]'
        server.ssl.certificate: "/etc/kibana/server.pem"
        server.ssl.enabled: 'true'
        server.ssl.key: "/etc/kibana/private/server.key"
#2

Hi @sknayar,

Are there any warnings or errors in the Kibana server log?

Can you also check the browser dev tools for any Javascript errors?

#3

Hi @chrisronline,
I don't see any warnings or errors in the server logs. I get a warning at the right bottom of Kibana screen momentarily as below. I haven't set any alerts


   To create an alert, set a value for
   xpack.encryptedSavedObjects.encryptionKey
   in your kibana.yml file. Learn how.

There are some java script errors - I'm not sure one about SAML idp is relevant. Our Kibana instance sits behind an apache proxy . Proxy does the IDP authentication.

  Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.

    bootstrap.js:43 ^ A single error about an inline script not firing due to content security policy is expected!
    home:1 Access to manifest at 'https://login.stanford.edu/idp/profile/SAML2/Redirect/SSO?SAMLRequest=fZLLbsIwEEV%2FJfKeOE6BCosgpbAoEi0RSbvopnLiAawaO%2FU4ffx9E0JVWJS175yZe%2BQpioOuedr4vdnAewPog6%2BDNsiPDwlpnOFWoEJuxAGQ%2B4rn6cOKx2HEa2e9rawmQYoIzitr5tZgcwCXg%2FtQFTxtVgnZe18jpxT0W4hemK11MgTZ0HyvytJq8PsQ0dKOG9NsnRckWLSHKCM65B9A250ylwgla9qesVUaTvMbkMpB5Wmer0mwXCTktRyzYQRsNJYs3lZwMxzFshKymsAEtjIWbQyxgaXp0D4hcRRHg2gyYKxgtzweccZeSJCd2t4pI5XZXVdT9iHk90WRDfpOz%2BDw2KcNkNm0E8yPi92Z8utY8euZzP61OqVn5H5NzR9b1HKRWa2q7yDV2n7OHQgPCWGEzvqRy28w%2BwE%3D&RelayState=ss%3Amem%3A438fa7ef9e334ad19b01c200adbc1bd9a99f857bf9ea49720370756c9da52861' (redirected from 'https://elk.stanford.edu/ui/favicons/manifest.json') from origin 'https://elk.stanford.edu' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
    login.stanford.edu/idp/profile/SAML2/Redirect/SSO?SAMLRequest=fZLLbsIwEEV%2FJfKeOE6BCosgpbAoEi0RSbvopnLiAawaO%2FU4ffx9E0JVWJS175yZe%2BQpioOuedr4vdnAewPog6%2BDNsiPDwlpnOFWoEJuxAGQ%2B4rn6cOKx2HEa2e9rawmQYoIzitr5tZgcwCXg%2FtQFTxtVgnZe18jpxT0W4hemK11MgTZ0HyvytJq8PsQ0dKOG9NsnRckWLSHKCM65B9A250ylwgla9qesVUaTvMbkMpB5Wmer0mwXCTktRyzYQRsNJYs3lZwMxzFshKymsAEtjIWbQyxgaXp0D4hcRRHg2gyYKxgtzweccZeSJCd2t4pI5XZXVdT9iHk90WRDfpOz%2BDw2KcNkNm0E8yPi92Z8utY8euZzP61OqVn5H5NzR9b1HKRWa2q7yDV2n7OHQgPCWGEzvqRy28w%2BwE%3D&RelayState=ss%3Amem%3A438fa7ef9e334ad19b01c200adbc1bd9a99f857bf9ea49720370756c9da52861:1 Failed to load resource: net::ERR_FAILED
    kbn-ui-shared-deps.js:453 INFO: 2020-09-11T17:25:15Z
      Adding connection to https://elk.stanford.edu/elasticsearch

    discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(columns:!(_source),filters:!(),index:'8a37bdd0-f37a-11ea-9b12-d1e0d0c847e4',interval:auto,query:(language:kuery,query:''),sort:!()):1 Access to internal resource at 'https://login.stanford.edu/idp/profile/SAML2/Redirect/SSO?SAMLRequest=fZLLboMwEEV%2FBXkfTKBRVCsg0WTRSGmDAu2im8rgSbBqbOoxffx9IaRqsmjWvnNm7pEXyBvVsrRztd7BewfovK9GaWTHh5h0VjPDUSLTvAFkrmJ5%2BrBhoR%2Bw1hpnKqOIlyKCddLopdHYNWBzsB%2BygqfdJia1cy0ySkG9%2Bei43hsrfBAdzWtZlkaBq31EQwduSLNtXhBv1R8iNR%2BQfwBlDlJfIqRoaX%2FGXio4ze9ASAuVo3m%2BJd56FZPXqqpEFPGSB%2FMoFAB7Xs4BQMzKmYgCzvsYYgdrPaBdTMIgDCbB7WQ6LaZzFs7YTfhCvOzU9k5qIfXhuppyDCG7L4psMnZ6BovHPn2AJItBMDsutmfKr2P5r2eS%2FGt1Qc%2FI45qWPfao9SozSlbfXqqU%2BVxa4A5iMiU0GUcuv0HyAw%3D%3D&RelayState=ss%3Amem%3Ad428c77c8995ec6520afa84224a1fcf97f095ec6bbcb86ae9132e3eab561a562' (redirected from 'https://elk.stanford.edu/ui/favicons/manifest.json') from origin 'https://elk.stanford.edu' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
    1.plugin.js:3 "#noDataReact" element has not been added to the DOM yet
    renderReact @ 1.plugin.js:3
    render @ 1.plugin.js:3
    (anonymous) @ 1.plugin.js:3
    $digest @ kbn-ui-shared-deps.js:414
    .....................................
    ....................................
    o.defer @ kbn-ui-shared-deps.js:414
    $evalAsync @ kbn-ui-shared-deps.js:414
    (anonymous) @ kbn-ui-shared-deps.js:414
    d @ kbn-ui-shared-deps.js:414
    then @ kbn-ui-shared-deps.js:414
#4

This sounds like https://github.com/elastic/kibana/issues/77130 which was recently fixed. The workaround is to set xpack.encryptedSavedObjects.encryptionKey in your kibana.yml to a string that is at least 32 characters long.

#5

yes, that worked. thank you!