New Kibana install but "urlForwarding" error after login

Elastic Stack Kibana
1

After following the ELK demo install I can login but get a page with an error

Version: 8.15.0
Build: 76360
Error: Definition of plugin "urlForwarding" not found and may have failed to load.
    at https://<mydomain>/kibana/<hash>/bundles/core/core.entry.js:16:321890

Environment overview:

  • Three node elasticsearch cluster (two data + voting_only)
  • All running on Ubuntu 24.04 minimal in an Ubuntu LXD host
  • The domain is in cloudflare using their "proxied dns" for SSL/IP obfuscation/etc
  • DMZed LXC running NGINX reverse proxy for kibana (and elastic which is running fine)
  • elastic is using self signed certs and kibana is configured to connect to it using local IPs
  • kibana connects to elastic using a unique api token
  • kibana is configured for basePath and the like (more below)

The configured options are

server.basePath: "/kibana"
server.rewriteBasePath: true
server.publicBaseUrl: "https://<domain>/kibana"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/http_ca.crt" ]
elasticsearch.ssl.verificationMode: none

Troubleshooting steps taken:

  • I've reinstalled kibana and even destroyed the container and recreated it from scratch.
  • I've tried different browsers:
    • Safari gets me to the "Welcome to Elastic" page but shows a toast message "an error occurred" but won't let me click on the "see full error" button
    • Chrome is the main problem which brought me here with the aforementioned error
    • I used the debug level of logging in kibana but no plugin errors were detected

Guesses:

  • Since this looks like an error in the JS on the page I was thinking maybe whatever build step creates the JS packages might not be doing the basePath part right?
  • Something to do with the mildly complex structure of the request:
    • CloudFlare proxied DNS > NGINX proxied connection > kibana?

Somewhere along the way I tried the curl from the Access Kibana docs and

root@kibana:~# curl -XGET 10.10.10.69:9200/
curl: (52) Empty reply from server

when I change the URL to https://10.10.10.68:9200 it complains of untrusted SSL cert (since it's self signed) not sure if this counts since I've copied over the CA cert and even told kibana to not verify certs.

Thank you for any feedback.

2

UPDATE2: I tried to add snapshot repo and monitoring and everything was failing to work. Eventually even a node went down. I got my cluster health back to green and in all that I discovered the docs about builtin_users and reset the password for kabana_system (I was using an api key before) and added that to the conig. Bit of a backslide since now the original error is back and now even Safari reports the "urlForwarding" error :frowning:

UPDATE1: After using Safari to move a long a bit (I started trying to setup a node for monitoring using Metricbeat) Chrome is now loading the welcome page. It too has the same error toast message that is unclickable.

All the setup tasks I did I think were on the tiebraker node and not in kibana but :person_shrugging:

3

It's working now?!?!

This morning I started in trying to bypass the cloudflare dns proxy SSL and path by creating a new subdomain and matching config in nginx, I was down that road when I thought I would just refresh the old page and see what it looks like... to my surprise it's working!

I think the biggest two changes were:

  1. patience; waiting some time for it to work itself out
  2. using the builtin user kibana_system in the kibana config
root@kibana:~# grep -v '^#' /etc/kibana/kibana.yml | grep -v '^
server.port: 5601
server.host: "10.10.10.60"
server.basePath: "/kibana"
server.rewriteBasePath: true
server.publicBaseUrl: "https://<domain>/kibana"
server.ssl.enabled: false
elasticsearch.hosts: ["https://10.10.10.69:9200","https://10.10.10.68:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "<password>"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/http_ca.crt" ]
elasticsearch.ssl.verificationMode: none
logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
pid.file: /run/kibana/kibana.pid

NOTE TO FUTURE READERS:
I'm gonna lean in on using the kibana_system user/pass as the solution. Maybe some other setting I tweaked along the way but I try to take notes on the steps, even the failed ones.

Dinosaur note from myself 20yrs ago while working with a large storage cluster: "walk away and don't look at it for 40 mins or so"... allowing clustered services to figure themselves out shouldn't take that long in this modern era, but patience is a virtue and communications can take some time to sort out.

The next steps

  1. Snapshots: I cannot register an S2 repo at the moment (I'm really pointing to backblaze's b2 for cost savings)
  2. Use config management like ansible or BASH scripts for all this so teardown/build up is repeatable.

Wish me luck :pray:

1 Like