Unable to install Fleet Server

Hello!

I am trying to install Fleet Server with my self-hosted ELK. So, I have the error:

sudo ./elastic-agent install  -f --fleet-server-es=https://****.org:9200   --fleet-server-service-token=AAEAAWV**** --fleet-server-policy=fleet-server-policy --fleet-server-es-insecure
{"log.level":"info","@timestamp":"2022-11-16T16:03:14.045Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-16T16:03:17.696Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: fleet-server-policy","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html

I've tried different dicisions from here and here. But nothing helps. I have policy "Fleet Server Policy" with Fleet Server integration.

Hope to your help. Thanks!

Hi @leschev,

Could you check if Elasticsearch is reachable? It could be that you have issue on the connection with ES.

curl -k -u {username}:{password} https://{ELASTICSEARCH_HOST}:9200

After checking the ES connection, could you also try adding the --insecure flag to your command?

Thanks,
Cristina

I made curl with -k cause I have problems with certs. But I start Fleet with --fleet-server-es-insecure, so I think it doesn't makes troubles for me.

root@fleet-server:~# curl -k -u ***:*** https://***:9200
{
  "name" : "elasticsearch-master-2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "5F88vkM4QjK2r2t4yUOJkw",
  "version" : {
    "number" : "8.4.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "2bd229c8e56650b42e40992322a76e7914258f0c",
    "build_date" : "2022-08-26T12:11:43.232597118Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Can you check the output of this API request? You can find the policy id if you look at the URL of Edit Fleet Server integration page.
https://<KIBANA_HOST:PORT>/api/fleet/package_policies/<id of fleet-server integration policy>

I would like to see if the inputs are generated correctly.

Sure.

{
  "item": {
    "id": "13bcfd99-a40a-4b46-9bd9-395f76a0f0fa",
    "version": "WzEyNjg1MywyXQ==",
    "name": "fleet_server-1",
    "namespace": "default",
    "description": "",
    "package": {
      "name": "fleet_server",
      "title": "Fleet Server",
      "version": "1.2.0"
    },
    "enabled": true,
    "policy_id": "fleet-server-policy",
    "output_id": "",
    "inputs": [
      {
        "type": "fleet-server",
        "policy_template": "fleet_server",
        "enabled": true,
        "streams": [],
        "vars": {
          "host": {
            "type": "text",
            "value": "172.24.54.23"
          },
          "port": {
            "value": [
              8220
            ],
            "type": "integer"
          },
          "max_agents": {
            "type": "integer"
          },
          "max_connections": {
            "type": "integer"
          },
          "custom": {
            "value": "",
            "type": "yaml"
          }
        },
        "compiled_input": {
          "server": {
            "port": 8220,
            "host": "172.24.54.23"
          }
        }
      }
    ],
    "revision": 1,
    "created_at": "2022-11-16T15:58:52.740Z",
    "created_by": "leschev",
    "updated_at": "2022-11-16T15:58:52.740Z",
    "updated_by": "leschev"
  }
}

I don't see anything wrong with the policy input.
What happens if you try the enroll without --fleet-server-es-insecure flag?

Also can you show the Settings page on the UI, I am curious is the Elasticsearch Output is set up correctly.

The same error.

sudo ./elastic-agent install -f  --fleet-server-es=https://****:9200   --fleet-server-service-token=AAEAA****   --fleet-server-policy=fleet-server-policy
{"log.level":"info","@timestamp":"2022-11-22T12:09:52.465Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-22T12:10:03.688Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":447},"message":"Retrying to restart...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-22T12:10:04.692Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: fleet-server-policy","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-22T12:10:06.695Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-22T12:10:10.696Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: fleet-server-policy","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html

Could you also check the result of this ES query, to see if the policy record contains the fleet-server input?

GET .fleet-policies/_search?q=fleet-server-policy

Could you also check the result of this ES query, to see if the policy record contains the fleet-server input?

pastebin

Also can you show the Settings page on the UI, I am curious is the Elasticsearch Output is set up correctly.

I don't see anything wrong in the output. Does the elasticsearch output host look good to you?

One thing you can try is resetting the fleet server policy:

curl --request POST
--url https://{YOUR_KIBANA}/internal/fleet/reset_preconfigured_agent_policies/fleet-server-policy
-u elastic:<your password>
--header 'Content-Type: application/json'
--header 'kbn-xsrf: debug'

All hosts in cluster are green. Nothing problems else.

Cluster we use are configured with SSO so my request is more complex, but I get 404:

curl 'https://{KIBANA}/internal/fleet/reset_preconfigured_agent_policies/fleet-server-policy' \
  -H 'authority: {KIBANA}' \
  -H 'accept: */*' \
  -H 'accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7' \
  -H 'content-type: application/json' \
  -H 'cookie: XXXX' \
  -H 'kbn-version: 8.4.1' \
  -H 'referer: https://{KIBANA}/app/fleet/policies' \
  -H 'sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Linux"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36' \
  -H 'x-kbn-context: %7B%22name%22%3A%22fleet%22%2C%22url%22%3A%22%2Fapp%2Ffleet%22%7D' \
  -H 'kbn-xsrf: debug' --compressed

And response:

{"statusCode":404,"error":"Not Found","message":"Not Found"}

How can I make it via UI? I've tried to remove and create fleet server policy earlier, but it is not helped me.

Since 8.4.0 we have a Debug UI in Fleet where you can trigger the Reset of preconfigured policies.
Though I am not sure if your policy is preconfigured, so this might not help.

https://KIBANA/app/fleet/_debug

Yes. There are not any options available here. I've deleted policy from this debug menu and generated it from quick start. Nothing changed. This is the new generated policy - pastebin

I had a look at the fleet-server logic and the contents of the .fleet-policies index sent earlier. According to the logic, fleet-server takes the latest revision_idx from the policies index, and I see 2 documents with latest revision 4, one of them doesn't contain a fleet-server input. I don't know how this could happen, but there is a workaround if this is the issue.

Can you try creating another Fleet Server policy with a different id, and try to use that to enroll a Fleet server?
You can pass it to the enroll command with this parameter --fleet-server-policy=cutom_id

What I have did:

  1. removed fleet-server-policy
  2. generated new policy with name Test Fleet Server Policy on this page: https://{KIBANA}/app/fleet/agents on "Advanced" tab
  3. tried to install fleet-server applied --fleet-server-policy=test-fleet-server-policy

And nothing changes :c The same error.

Hi @leschev,

I suspect that there might be a problem in the command you used to install the fleet server. I tried locally using the same command and failed for me with "context canceled". Could you try with this instead?

sudo ./elastic-agent install -f \
  --fleet-server-es=http://localhost:9200 \
  --fleet-server-service-token=**** \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-es-insecure

The difference is that it's exposing the ES connection on http rather than https. Let us know if this works for you.

Hello!

I've tried to change https to http:

./elastic-agent install -f  --fleet-server-es=http://****:9200   --fleet-server-service-token=****   --fleet-server-policy=test-fleet-server-policy
{"log.level":"info","@timestamp":"2022-11-23T14:45:11.193Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-23T14:45:14.961Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Error - EOF","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html

With --fleet-server-es-insecure the same result.

Could you also change the "Elasticsearch Host" in outputs to http protocol and then re run the command? They need to be the same in order to work properly.

Sure. I've changed it to http and tried. The same error with EOF.

You can make sure. This is settings page from UI:

Hi @leschev,

it's a bit difficult to diagnose your issue without having more info.

Could you share some more details around your configuration (like the versions of the installed stack, how they were installed etc) and also your kibana.yml?