I upgraded the whole ELK stack from 7.3.x to 7.4.0 yesterday. After upgrade, I am unable to open Kibana from the browser after authentication. It goes into a loop refreshing the page with errors all the time. Below is the screenshot of the error that I get.
I executed the same curl command, and I got the file size as 22868674. But the actual file size is of the vendors.bundle.dll.js is 22868683.
I am unable to download the whole file even though I am executing the curl command on the same host where kibana is installed. But Kibana is setup to run on the private IP address of the host rather than the localhost.
I do not have any custom plugins installed. I upgraded the kibana version from 7.3.2 to 7.4.1 using the elastisearch rpm package distribution. Prior to the upgrade it was working fine without any issues. I had even updated from 7.1 to 7.2 and then 7.2 to 7.3 as well using the yum command and kibana upgraded and started just fine.
Also I have upgraded the whole stack to 7.4.1 except for Beats.
Hey @zulfiqar4891 thanks for the HAR and log. The HAR contains a few requests which look unusual....
For example, the request to https://tapro.altayyargroup.com:5601/bundles/af7ae505a9eed503f8b8e6982036873e.woff2 has a response status of 0.
Do you have a reverse-proxy in front of Kibana? If so, could you try accessing Kibana not going through the reverse-proxy to see if you still get the infinite digest loops?
@Brandon_Kobel,
Kibana is hosted on an ec2 instance on AWS and is accessible through a load balancer to public network. But, I see the same issue appearing even if I access kibana directly using the hosting server's private IP address bypassing the load balancer from inside the VPC.
@zulfiqar4891 are you always seeing this digest cycle loop when trying to use Kibana? Are you using security where you have to login before being able to use Kibana; and if so, are you able to login at least?
Also, to rule out an issue with the upgrade potentially leaving some old files behind that is causing this, can you try running sudo yum remove kibana and ensuring that /usr/share/kibana is empty before re-running sudo yum install kibana?
@Brandon_Kobel,
What do you mean by digest cycle loop? Do you mean kibana constantly trying to refresh but keeps failing with the error?
Yes. I do have security enabled. And I think I am able to login successfully But just that I am unable to proceed further.
Won't removing kibana delete all my created visualizations, dashboards and saved searches? Or are my visualizations, dashboards and other stuff are stored on elastisearch?
Let me know your thoughts.
And thank you very much for your support in helping me resolve this. Really appreciate your help
A large number of resources are loaded, and then something is happening where the browser continues to request /app/kibana, but all requests are aborted.
The error that is being thrown 10 $digest() iterations reached. Aborting! is an AngularJS thing called the "digest cycle" which is when AngularJS detects an endless loop.
Won't removing kibana delete all my created visualizations, dashboards and saved searches? Or are my visualizations, dashboards and other stuff are stored on elastisearch?
All of your dashboard, visualizations and saved-searches are stored in Elasticsearch, so these won't be list. If you've heavily modified your /etc/kibana/kibana.yml file, it might be worth backing it up before the yum remove kibana, but I believe it should be left there during the remove operation.
Thanks for the info @Brandon_Kobel. I will completely uninstall kibana and then do a fresh install. I haven't changed much on the kibana config file except for host, elastisearch host and the kibana username and password for elastisearch. I'll back it up anyways and then do a complete new install tomorrow. Will keep here posted.
The same issue still exists event after completely removing Kibana through yum remove command, deleting the contents of the usr/share/kibana and doing a fresh install.
I even updated the whole stack to version 7.4.1 from 7.4.0. And still the same issue. With version 7.3.2, everything was working fine.
Below are the 2 errors that I see on the browser console and the Kibana server logs.
> # Kibana is served by a back end server. This setting specifies the port to use.
> #server.port: 5601
>
> # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
> # The default is 'localhost', which usually means remote machines will not be able to connect.
> # To allow connections from remote users, set this parameter to a non-loopback address.
> server.host: "172.20.3.177"
>
> # Enables you to specify a path to mount Kibana at if you are running behind a proxy.
> # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
> # from requests it receives, and to prevent a deprecation warning at startup.
> # This setting cannot end in a slash.
> #server.basePath: ""
>
> # Specifies whether Kibana should rewrite requests that are prefixed with
> # `server.basePath` or require that they are rewritten by your reverse proxy.
> # This setting was effectively always `false` before Kibana 6.3 and will
> # default to `true` starting in Kibana 7.0.
> #server.rewriteBasePath: false
>
> # The maximum payload size in bytes for incoming server requests.
> #server.maxPayloadBytes: 1048576
>
> # The Kibana server's name. This is used for display purposes.
> #server.name: "your-hostname"
>
> # The URLs of the Elasticsearch instances to use for all your queries.
> elasticsearch.hosts: ["http://172.20.3.177:9200"]
>
> # When this setting's value is true Kibana uses the hostname specified in the server.host
> # setting. When the value of this setting is false, Kibana uses the hostname of the host
> # that connects to this Kibana instance.
> #elasticsearch.preserveHost: true
>
> # Kibana uses an index in Elasticsearch to store saved searches, visualizations and
> # dashboards. Kibana creates a new index if the index doesn't already exist.
> #kibana.index: ".kibana"
>
> # The default application to load.
> kibana.defaultAppId: "tapro2"
>
> # If your Elasticsearch is protected with basic authentication, these settings provide
> # the username and password that the Kibana server uses to perform maintenance on the Kibana
> # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
> # is proxied through the Kibana server.
> elasticsearch.username: "kibana"
> elasticsearch.password: "#########"
>
> # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
> # These settings enable SSL for outgoing requests from the Kibana server to the browser.
> #server.ssl.enabled: false
> #server.ssl.certificate: /path/to/your/server.crt
> #server.ssl.key: /path/to/your/server.key
>
> # Optional settings that provide the paths to the PEM-format SSL certificate and key files.
> # These files validate that your Elasticsearch backend uses the same key files.
> #elasticsearch.ssl.certificate: /path/to/your/client.crt
> #elasticsearch.ssl.key: /path/to/your/client.key
>
> # Optional setting that enables you to specify a path to the PEM file for the certificate
> # authority for your Elasticsearch instance.
> #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
>
> # To disregard the validity of SSL certificates, change this setting's value to 'none'.
> #elasticsearch.ssl.verificationMode: full
>
> # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
> # the elasticsearch.requestTimeout setting.
> #elasticsearch.pingTimeout: 1500
>
> # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
> # must be a positive integer.
> #elasticsearch.requestTimeout: 30000
>
> # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
> # headers, set this value to [] (an empty list).
> #elasticsearch.requestHeadersWhitelist: [ authorization ]
>
> # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
> # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
> #elasticsearch.customHeaders: {}
>
> # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
> #elasticsearch.shardTimeout: 30000
>
> # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
> #elasticsearch.startupTimeout: 5000
>
> # Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
> #elasticsearch.logQueries: false
>
> # Specifies the path where Kibana creates the process ID file.
> #pid.file: /var/run/kibana.pid
>
> # Enables you specify a file where Kibana stores log output.
> logging.dest: /var/log/kibana/kibana.log
>
> # Set the value of this setting to true to suppress all logging output.
> #logging.silent: false
>
> # Set the value of this setting to true to suppress all logging output other than error messages.
> #logging.quiet: false
>
> # Set the value of this setting to true to log all events, including system usage information
> # and all requests.
> #logging.verbose: false
>
> # Set the interval in milliseconds to sample system and process performance
> # metrics. Minimum is 100ms. Defaults to 5000.
> #ops.interval: 5000
>
> # Specifies locale to be used for all localizable strings, dates and number formats.
> # Supported languages are the following: English - en , by default , Chinese - zh-CN .
> #i18n.locale: "en"
Thanks a lot for your help @Brandon_Kobel. This fixed the issue. Kibana loads up without any issues now.
In the process of reverting the version to 7.3.2, I was getting kibana index migration errors due to which I had to delete the .kibana_N indices and this deleted some of my visualisations and dashboards. Now I will have to recreate them all again. But I am glad that the issue is resolved now.
Thank you again for your help and support. Really appreciate it.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
