Fleet not working anymore [Unable to initialize Fleet]

Hello,

I am trying elastic Endpoint under elastic and kibana version 8.0.0 (from source), and since few days I am geeting this error in my logs when I click on the Fleet on Kibana

404 Not Found' error response from package registry at https://epr-snapshot.elastic.co/package/endpoint/0.17.0-dev.3/

and here is the error on Kibana interface:

Could you tell me please why that link is not anymore reacheable ? and how to solve that please !

Thanks

Hi, I am not sure myself, but the Fleet team has seen this happen when the package is removed from the registry. I am sorry for the inconvenience! I will confer and we will post back how to fix and figure it out.

2 Likes

Thanks for your answer @EricDavisX

You can check the registry to see what the current package version is by looking at the registry directly with

https://epr-snapshot.elastic.co/search?experimental=true (for all packages in their latest version)

and

https://epr-snapshot.elastic.co/search?experimental=true&package=symantec&all=true (for all available versions of the symantec package, which is indeed only one, and not the one you have installed).

You can use this information to update your symantec package to the latest version with this curl command:

curl -X POST -u $USER:$PASSWORD http://$KIBANA_HOST:$KIBANA_PORT/$KIBANA_BASEPATH/api/fleet/epm/packages/symantec-0.1.2 -H 'kbn-xsrf: xyz'

This is the same api endpoint the UI would use to install or update the package, if you weren't blocked by the error.

$KIBANA_BASEPATH is a random three-letter combination set when you run kibana in dev mode from source with yarn start, if this is not the case for you, leave it out. You can also look into the network tab of the browser dev console and filter for paths containing api/fleet/epm to find the correct base URL to use.

Thank you for testing fleet, and apologies for the inconvenience!

You can do the same for the endpoint package:

https://epr-snapshot.elastic.co/search?experimental=true&package=endpoint&all=true

shows the latest version available is endpoint-0.17.0-dev.6.

Out of curiosity, did you install the symantec package on your system, or did that error appear out of the blue?

Thanks for your answer @skh,

When I tried in the browser $KIBANA_IP:$KIBANA_PORT/api/fleet/epm, I got an 404 error so I didn't know what to put in the $KIBANA_BASEPATH field

{"statusCode":404,"error":"Not Found","message":"Not Found"}

And when I run NODE_OPTIONS="--no-warnings" yarn start --run-examples I don't see the three letters that you spoke about.
here is the log that I am getting when I start kibana:

yarn run v1.22.5
$ node scripts/kibana --dev --run-examples
[2020-12-17T13:39:03.585Z][INFO ][plugins-service] Plugin initialization disabled.
[2020-12-17T13:39:03.614Z][WARN ][savedobjects-service] Skipping Saved Object migrations on startup. Note: Individual documents will still be migrated when read or written.
 no-base-path  ====================================================================================================
 no-base-path  Running Kibana in dev mode with --no-base-path disables several useful features and is not recommended
 no-base-path  ====================================================================================================
 watching for changes  (8024 files)
np bld    log   [14:39:06.668] [info][@kbn/optimizer] initialized, 120 bundles cached
np bld    log   [14:39:06.672] [warning][@kbn/optimizer] only building [v7dark,v7light] themes, customize with the KBN_OPTIMIZER_THEMES environment variable
np bld    log   [14:39:06.673] [success][@kbn/optimizer] all bundles cached, success after 2.2 sec
 server    log   [14:39:11.705] [info][plugins-service] Plugin "visTypeXy" is disabled. 
 server    log   [14:39:11.970] [info][plugins-system] Setting up [107] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,routingExample,translations,share,legacyExport,embeddable,uiActionsEnhanced,esUiShared,expressions,charts,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVega,visTypeVislib,visTypeTimelion,features,licenseManagement,dataEnhanced,watcher,canvas,visTypeTimeseries,visTypeTimeseriesEnhanced,visTypeTagcloud,visTypeMetric,visTypeTable,visTypeMarkdown,tileMap,regionMap,mapsOss,lensOss,inputControlVis,graph,timelion,dashboard,embeddableExamples,dashboardEnhanced,visualize,stateContainersExamples,searchExamples,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,encryptedSavedObjects,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,ml,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,securitySolution,case,infra,monitoring,logstash,apm,alertingExample,uptime,bfetchExplorer] 
 server    log   [14:39:11.972] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: c52080a2-0f7f-44b8-981b-79e85e7c0955 
 server    log   [14:39:12.208] [info][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox. 
 server    log   [14:39:12.350] [info][monitoring][monitoring][plugins] config sourced from: production cluster 
 server    log   [14:39:12.552] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations... 
 server    log   [14:39:12.771] [info][savedobjects-service] Starting saved objects migrations 
 server    log   [14:39:12.928] [info][plugins-system] Starting [107] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,routingExample,translations,share,legacyExport,embeddable,uiActionsEnhanced,esUiShared,expressions,charts,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVega,visTypeVislib,visTypeTimelion,features,licenseManagement,dataEnhanced,watcher,canvas,visTypeTimeseries,visTypeTimeseriesEnhanced,visTypeTagcloud,visTypeMetric,visTypeTable,visTypeMarkdown,tileMap,regionMap,mapsOss,lensOss,inputControlVis,graph,timelion,dashboard,embeddableExamples,dashboardEnhanced,visualize,stateContainersExamples,searchExamples,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,encryptedSavedObjects,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,ml,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,securitySolution,case,infra,monitoring,logstash,apm,alertingExample,uptime,bfetchExplorer] 
 server    log   [14:39:13.962] [info][listening] Server running at https://10.10.13.135:5601 
 server    log   [14:39:14.597] [info][server][Kibana][http] http server running at https://10.10.13.135:5601

And answering your question about symantec, I didn't install it, I am just trying the Endpoint for the moment.

  1. BASEPATH

It seems that --run-examples disables the base path, so it's as if you'd run yarn start --no-base-path, as you see from the log output. You don't need --run-examples, but you can continue to do so for now, and just assume BASEPATH is empty

  1. $KIBANA_IP:$KIBANA_PORT/api/fleet/epm

This is not a valid API endpoint, but $KIBANA_IP:$KIBANA_PORT/api/fleet/epm/packages should return something. Does this work for you?

  1. Browser vs. curl

You will need to send an http POST request, so this will not work in the browser, you'll need to use curl as in the example I posted above.

Another question: as you're running master from source, I assume this is not a production environment. It might be easier to start over from scratch, i.e. with no data in elasticsearch.

I tried $KIBANA_IP:$KIBANA_PORT/api/fleet/epm/packages and it worked, and I am getting some output, an example for the endpoint :

name: "endpoint
title: "Endpoint Security"
version: "0.17.0.dev.6"
release: "beta"
description: "Protect your hosts with …curity data visibility."
type: "integration"
download: "/epr/endpoint/endpoint-0.17.0-dev.6.zip"
path: "/package/endpoint/0.17.0-dev.6"

But there is no information about symantec

And yes it's not a production environment, I can delete all the data inside, but I prefer to let this as a last possibility if there is no way to solve this problem.

and just a question, in the case I start from the scratch, should I just delete all the index, or I have to rebuild elasticsearch and kibana to solve this issue ?

In that case, best try to update the endpoint package with the curl command I gave above.

If that doesn't help, starting from scratch might be your only option. Please bear in mind that on master, things might break at any time.

Moving away or deleting the data folder that is used by elasticsearch and restarting elasticsearch and kibana should do it.

I tried to update the endpoint package using :

curl -X POST -k -u elastic:password https://X.X.X.X:5601/api/fleet/epm/packages/endpoint-0.17.0-dev.6 -H 'kbn-xsrf: xyz'

{"statusCode":400,"error":"Bad Request","message":"endpoint-0.17.0-dev.6 is out-of-date and cannot be installed or updated"}

and if I try without -dev.6 I get symantec error:

{"statusCode":502,"error":"Bad Gateway","message":"'404 Not Found' error response from package registry at https://epr-snapshot.elastic.co/package/symantec/0.1.0/"}

I will try to delete everything later and see if it works, I will keep you updated
Thanks for all your answers and help :slight_smile:

1 Like

As last 2 questions before I delete everything :sweat_smile:
Is there a way to uninstall the endpoint package and install it again from a curl request ?

and the seconde question: I have tried to force installing a previous version and then force upgrade as it's mentionned here (tested that for system package like he did just for test) :

But when I run POST request in the dev tools I get this error !!

{
  "error" : "no handler found for uri [/api/fleet/epm/packages/system-0.8.2?pretty=true] and method [POST]"
}

You can't use Kibana dev tools to send a POST request to a Kibana server API endpoint.
You really need to use curl from the command line, or another tool like postman.

Kibana dev tools send all requests to elasticsearch.
In this case, the API endpoint you need to call is in Kibana Server, NOT elasticsearch.

endpoint is a mandatory package, so you won't be able to uninstall it.

1 Like

Thanks for your answers @skh ^ ^

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.