Broken Fleet setup

Elastic Security Endpoint Security
1

Hello!
I recently wanted to setup Fleet on a working ELK cluster (self managed).
By following this guide I did the initial steps: Add a Fleet Server | Fleet and Elastic Agent Guide [7.17] | Elastic
(downloaded the package, ran it and started to fill out the form in Kibana)
But somewhere in the middle of the setup I upgraded ELK from 7.16.3 to 7.17 without finishing the setup before upgrading (because I forgot that I started the setup).
After the upgrade I cannot access KIbana>Fleet section, it loads for a long time and puts out an error in KIbana:

Unable to initialize Fleet

<!DOCTYPE html><html lang="en"><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width, initial-scale=1"><ti$

KIbana logs seems to contain these entries:

{"type":"log","@timestamp":"2022-02-04T17:37:46+02:00","tags":["warning","plugins","fleet"],"pid":637280,"message":"Failed installing package [endpoint] due to error: [TimeoutError: Request timed out]"}
[2022-02-04T17:37:46.457+02:00][WARN ][plugins.fleet] Failed installing package [endpoint] due to error: [TimeoutError: Request timed out

Is it possible to restart the setup (delete some Fleet related data)?
Or any other recommendations where to look what is causing this error?

2

@queried1

Thanks for trying out Fleet, I'm sorry it's not working right now.

The error you're seeing makes me think that that there may be an issue with your connection to the Elastic Package Registry. From the logs, Fleet is attempting to install the Endpoint package from the package registry and is timing out. The Endpoint package is installed automatically if you've ever used the Security application.

With your self managed setup, are you running your own package registry? Or are you using a proxy? You can read more about different options connecting to the package registry here: Air-gapped environments | Fleet and Elastic Agent Guide [7.17] | Elastic

It's possible your stack isn't able to connect to the registry properly.

Another troubleshooting step you can take is to navigate to the Integrations app. This app will list all Integrations/packages available from the package registry. If you're able to see a list of integrations like in the screenshot below, it's likely that you're able to access the registry. Get to this page by clicking the top left menu icon and click "Add integrations".

image
image2880×1580 496 KB

Finally, could you provide your Kibana and ES logs after trying to visiting Fleet? There may be another reason that Fleet can't install the package that is being masked by the timeout message in the logs. If you'd like to share the logs privately, I can share my email.

3

Thank you for the fast reply!
I spent the day trying to find something useful.

I am using a reverse proxy and found that directly accessing KIbana's port allows me to see the Fleet page (probably a timeout issue for Apache that I will try to fix later). Now I can access the Fleet section, but KIbana does show the following logs:

{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.020+02:00","message":"Found previous transform references:\n [{\"id\":\"endpoint.metadata_current-default-1.2.2\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.2.2\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.2.2\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.2.2\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.2.2\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.2.2\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_current-default-1.3.0\",\"type\":\"transform\"},{\"id\":\"endpoint.metadata_united-default-1.3.0\",\"type\":\"transform\"}]","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.020+02:00","message":"Deleting currently installed transform ids endpoint.metadata_current-default-1.2.2,endpoint.metadata_united-default-1.2.2,endpoint.metadata_current-default-1.2.2,endpoint.metadata_united-default-1.2.2,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.2.2,endpoint.metadata_united-default-1.2.2,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0,endpoint.metadata_current-default-1.3.0,endpoint.metadata_united-default-1.3.0","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.074+02:00","message":"Deleted: endpoint.metadata_current-default-1.2.2","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.074+02:00","message":"cannot find transform for endpoint.metadata_current-default-1.2.2","log":{"level":"WARN","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.075+02:00","message":"Deleted: endpoint.metadata_united-default-1.2.2","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.075+02:00","message":"cannot find transform for endpoint.metadata_united-default-1.2.2","log":{"level":"WARN","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.075+02:00","message":"Deleted: endpoint.metadata_current-default-1.2.2","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.077+02:00","message":"cannot find transform for endpoint.metadata_current-default-1.2.2","log":{"level":"WARN","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.078+02:00","message":"Deleted: endpoint.metadata_united-default-1.2.2","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.078+02:00","message":"cannot find transform for endpoint.metadata_united-default-1.2.2","log":{"level":"WARN","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.079+02:00","message":"Deleted: endpoint.metadata_current-default-1.2.2","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.079+02:00","message":"cannot find transform for endpoint.metadata_current-default-1.2.2","log":{"level":"WARN","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.088+02:00","message":"Deleted: endpoint.metadata_united-default-1.2.2","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":451246}}
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:36.088+02:00","message":"cannot find transform for endpoint.metadata_united-default-1.2.2","log":{"level":"WARN","logger":"plugins.fleet"},"process":{"pid":451246}}
####Many of these below
{"ecs":{"version":"1.12.0"},"@timestamp":"2022-02-08T17:07:46.413+02:00","message":"Deleted: endpoint.metadata_united-default-1.3.0","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":451246}}

This error gave a hint to look into the Integrations section as you suggested too.
I can access the Integration section and I can see the Endpoint Security integration as well. There I found an issue that I was not able to update the Endpoint Security package to 1.3.0, it just kept going back to the old version.

The errors hinted something about transforms, I check those and they seemed to be working, but I decided to set "transform" role to two of my least used nodes. Removed the Fleet server integration as a thought that it may trigger something and probably did turn on and off all possible settings found under Integrations and Transforms sections.

Yesterday I accessed the Integrations>Endpoint Security>Settings section directly using the Kibana's port without the reverse proxy (Apache), pressed update the package and left the page open till next morning. Today I can see that the package has updated and does not jump back to the old version, no more errors logs and I can access the Fleet section instantly, even added a Fleet server, though had some trouble selecting a policy (the drop down menu was empty), but after messing the policy settings and manually settings the policy's ID in the command line I was able to add the Fleet server. There were old entries under the Fleet>Agents section, removed those and everything seems to be working properly.

My conclusion is that the reverse proxy did not allow some kind of a background process to finish properly and after messing with some settings I allowed Kibana to finish a task.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.