OpenTelemetry Integration on AirGapped Environment

Elastic Stack Elastic Agent
1

Hi, we have two ELK clusters, one for production and other for monitoring.

The cluster don’t have access to internet, so is an AirGapped environment, the Elastic Artifacts are being rerouted using Sonatype Nexus Repository as Agent Binary Download, and the Elastic Package Registry is been hosted on OpenShift, using the “distribuition” version.

I found an Issue when trying to deploy an Integration of OpenTelemetry Collector, it give the error: [system_otel] package not installed of found in registry

Looking for this, I think the Kibana/Fleet is trying to access the otel schema on OpenTelemetry.io or in GitHub.com which has the latest schema in there.

Said that, if this is true, how can I configure the fleet/kibana to use Nexus to access the data?
Are any plans to host it direct on Elastic Artifacts since I think this OTel Integration is in fact EDOT Collector?

2

I was hosting the docker.elastic.co/package-registry/distribution:9.3.0 and I tried to change it to docker.elastic.co/package-registry/distribution:production and the https://epr.elastic.co/package/system_otel/0.2.5/ don’t exist in both images.

I tried to use Sonatype Nexus Repository to proxy the EPR, but it’s performance in this case is a bit awful.

I will continue to try to find a way, since the EPR images have 33GB.

3

Okay, I was able to use Nexus to proxy the epr.elastic.co, the performance issue was the Kibana was complaining about the internal certificate, which I resolved using NODE_EXTRA_CA_CERTS, this can be set on sysconfig or overriding kibana systemd config.

Now I am getting other issue.

image
image311×166 7.82 KB

It only show this error when on the Overview tab, other tabs appears are working

image
image889×367 21.6 KB

Maybe the problem is with Nexus, I need to investigate, but the weird part is Kibana showing text/x-yaml instead of application/x-yaml which is the standard now since text/x-yaml is not UTF-8 complaint.

4

The weirdness didn’t stopped.

{"service":{"node":{"roles":\["background_tasks","ui"\]}},"ecs":{"version":"9.0.0"},"@timestamp":"2026-02-20T13:26:56.920-03:00","message":"'404 Not Found' error response from package registry at https://nexus/repository/epr-elastic/package/threat_map/1.34.0/changelog.yml","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":784662,"uptime":76878.851975498},"trace":{"id":"05a5605deeabe2b1c832aaa5658f5186"},"transaction":{"id":"d9e2b75b2a3645a2"}}
{"service":{"node":{"roles":\["background_tasks","ui"\]}},"ecs":{"version":"9.0.0"},"@timestamp":"2026-02-20T13:26:58.970-03:00","message":"'404 Not Found' error response from package registry at https://nexus/repository/epr-elastic/package/threat_map/1.34.0/changelog.yml","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":784662,"uptime":76880.902096593},"trace":{"id":"c598dc562cc3d66fe38f848b97cd7b4a"},"transaction":{"id":"07731c01780ee4e9"}}
{"service":{"node":{"roles":\["background_tasks","ui"\]}},"ecs":{"version":"9.0.0"},"@timestamp":"2026-02-20T13:35:48.910-03:00","message":"getPackage - otel_internal_telemetry-1.34.0 failed with error: PackageNotFoundError: otel_internal_telemetry@1.34.0 not found","log":{"level":"WARN","logger":"plugins.fleet"},"process":{"pid":784662,"uptime":77410.843090167},"trace":{"id":"d637e85b9d12d0248a0cc1fb02461e8e"},"transaction":{"id":"2819774922a5d233"}}
{"service":{"node":{"roles":\["background_tasks","ui"\]}},"ecs":{"version":"9.0.0"},"@timestamp":"2026-02-20T13:35:48.912-03:00","message":"otel_internal_telemetry@1.34.0 not found","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":784662,"uptime":77410.843482981},"trace":{"id":"d637e85b9d12d0248a0cc1fb02461e8e"},"transaction":{"id":"2819774922a5d233"}}
{"service":{"node":{"roles":\["background_tasks","ui"\]}},"ecs":{"version":"9.0.0"},"@timestamp":"2026-02-20T13:35:48.918-03:00","message":"getPackage - otel_internal_telemetry-1.34.0 failed with error: PackageNotFoundError: otel_internal_telemetry@1.34.0 not found","log":{"level":"WARN","logger":"plugins.fleet"},"process":{"pid":784662,"uptime":77410.849554084},"trace":{"id":"5b48d2de08ff51ef526f877a59fa2b43"},"transaction":{"id":"ef4382cc7f68b165"}}
{"service":{"node":{"roles":\["background_tasks","ui"\]}},"ecs":{"version":"9.0.0"},"@timestamp":"2026-02-20T13:35:48.918-03:00","message":"otel_internal_telemetry@1.34.0 not found","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":784662,"uptime":77410.849860826},"trace":{"id":"5b48d2de08ff51ef526f877a59fa2b43"},"transaction":{"id":"ef4382cc7f68b165"}}
{"service":{"node":{"roles":\["background_tasks","ui"\]}},"ecs":{"version":"9.0.0"},"@timestamp":"2026-02-20T13:36:01.931-03:00","message":"getPackage - system_otel-1.34.0 failed with error: PackageNotFoundError: system_otel@1.34.0 not found","log":{"level":"WARN","logger":"plugins.fleet"},"process":{"pid":784662,"uptime":77423.863319837},"trace":{"id":"c622538b8c126cf4262f2bace52a2089"},"transaction":{"id":"5900c6bb4129700f"}}
{"service":{"node":{"roles":\["background_tasks","ui"\]}},"ecs":{"version":"9.0.0"},"@timestamp":"2026-02-20T13:36:01.932-03:00","message":"system_otel@1.34. not found","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":784662,"uptime":77423.864080052},"trace":{"id":"c622538b8c126cf4262f2bace52a2089"},"transaction":{"id":"5900c6bb4129700f"}}

The OpenTelemetry integration is on 0.2.5, then why Kibana is trying to access the 1.34.0? Looking for 1.34.0 on EPR(search), I found 1password and Network Traffic using this specific version, maybe it are not parsing the correct version? A few Integrations show this issue while navigating, but appears downloading it is not a problem, because I can see in Nexus and on Integration Installed, they are there.

Just adding the OpenTelemetry still are not operational, it show some instruction to download manually while showing the system_otel not found then I don't know if the procedure is true manually or if Elastic Agent can or can't deply an Elastic Collector and manage the pipeline from Kibana interface.