Kibana : Fleet & Integrations infinite loading on new Air Gap installation

Elastic Stack Kibana
,
1

TL;TR

In an AirGap environment, Kibana, "Management > Fleet & Integrations" shows an infinite loading despite successful connection to the local EPR according to the logs. No errors in browser and no obvious errors in Kibana logs.

Hello,

I'm having trouble with the installation of Elasticsearch + Kibana + Fleet on a fresh Debian 12.9 machine in an Air Gapped environment (w/o internet access).

I followed the instructions from the documentation, and Air Gap installation instructions.

Here's exactly what I have done :

  • Installed Elasticsearch and Kibana from the .deb and enrolled Kibana with Elasticsearch
  • Generated self-signed certificates for HTTPS communications with local EPR stored in /etc/kibana/certs with read permissions for the kibana user.
    • /usr/share/elasticsearch/bin/elasticsearch-certutil cert --self-signed --dns "kibana.local,epr.elastic.co,epr.local,kibana-knowledge-base-artifacts.elastic.co,kibana-kb.local,artifacts.local" --ip "127.0.0.1,192.168.1.20" --name "EPR local" --pem -out "epr.zip"
  • Downloaded the EPR container image docker.elastic.co/package-registry/distribution:lite-8.17.3 and ran it with Podman
    • podman run -it --rm --name EPR -p 443:443 -v /etc/kibana/certs/elk_local.key:/etc/ssl/elk_registry.key:ro -v /etc/kibana/certs/elk_local.crt:/etc/ssl/elk_registry.crt:ro -e "EPR_ADDRESS=0.0.0.0:443" -e "EPR_TLS_KEY=/etc/ssl/elk_registry.key" -e "EPR_TLS_CERT=/etc/ssl/elk_registry.crt" docker.elastic.co/package-registry/distribution:lite-8.17.3
    • Added the line Environment=NODE_EXTRA_CA_CERTS=/etc/kibana/certs/elk_local.crt to Kibana service file in /lib/systemd/system/kibana.service.
  • Downloaded the Artifacts for the EAR (Elastic Artifacts Registry) and hosted it with a python webserver module
    • python3 -m http.server -d artifacts_dir/ 8888
  • Modified /etc/hosts to add epr.local pointing to 127.0.0.1
  • Added to kibana.yml the following lines : 
    •   xpack.fleet.isAirGapped: true
  xpack.fleet.registryUrl: "https://epr.local"
  newsfeed.enabled: false
  telemetry.optIn: false
  • Reloaded Elasticsearch and Kibana services
  • Checked the logs for errors

Technical Stack

  • Debian 12.9 (with no network configured)
  • Kibana, Elasticsearch, Fleet 8.17.3
  • Elastic Package Repository (EPR) Lite container image 8.17.3
  • Artifacts Registry (Elastic Agents) 8.17.3

Testing and observations

Tested with :

  • curl --cacert /etc/kibana/certs/elk_local.crt https://epr.local
    => returns the same as https://epr.elastic.co/
  • curl -v http://localhost:8888 => ok

EPR container logs show a sucessfull request from Kibana (200 status).

No obvious errors in browser console (unless the one that according to the code is expected).

The only error I grep in Kibana logs is related to documentation update, and doesn't happen in 8.15.5 (see "Other Tests").

[ERROR][plugins.taskManager] Task ProductDocBase:EnsureUpToDate "ProductDocBase:EnsureUpToDate" failed: FetchError: request to https://kibana-knowledge-base-artifacts.elastic.co/?max-keys=1000 failed, reason: getaddrinfo EAI_AGAIN kibana-knowledge-base-artifacts.elastic.co

Other Tests

  • Used containers to deploy Elasticsearch + Kibana + EPR in the same network and updated configuration. It didn't worked, same issue.
  • Downgraded to 8.15.5, same issue.
  • Checked for conficting services on network ports, none found.

Question

I tried a lot of things to make it work but I don't see where I made a mistake. Do you have any suggestions ?

Thank you

2

Update

I found a way to solve the problem. It looks like Kibana cannot run Fleet if a network interface other than lo (or a bridged interface on it like podman0) is not up and active. In other words, you need to have a network interface with an ethernet cable plugged in to make it works.

How I found it out ?

In a VM, I installed and configured the latest version (8.17.4) of Elasticssearch, Kibana and EPR with containers via Podman.
In Virtualbox, I setup the network interface to be "disconnected".
When I open Kibana, the infinite loading in "Management > Fleet & Integrations" occurs BUT as soon as tell Virtualbox to "connect the network interface" in "Host-Only" mode (= without internet), the Fleet and Integrations page loads successfully !

On my physical machine, I plugged an ethernet cable between my two interfaces, which make them active and it works, problem solved (temporary).

But damm, that was hard to find, no errors, no relevant logs, no timeout, how could I've known that ?? I suspect it to be a bug.