Unable to install Fleet Server

leschev · November 16, 2022, 4:13pm

Hello!

I am trying to install Fleet Server with my self-hosted ELK. So, I have the error:

sudo ./elastic-agent install  -f --fleet-server-es=https://****.org:9200   --fleet-server-service-token=AAEAAWV**** --fleet-server-policy=fleet-server-policy --fleet-server-es-insecure
{"log.level":"info","@timestamp":"2022-11-16T16:03:14.045Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-16T16:03:17.696Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: fleet-server-policy","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html

I've tried different dicisions from here and here. But nothing helps. I have policy "Fleet Server Policy" with Fleet Server integration.

Hope to your help. Thanks!

Cristina_Amico · November 17, 2022, 10:06am

Hi @leschev,

Could you check if Elasticsearch is reachable? It could be that you have issue on the connection with ES.

curl -k -u {username}:{password} https://{ELASTICSEARCH_HOST}:9200

After checking the ES connection, could you also try adding the --insecure flag to your command?

Thanks,
Cristina

leschev · November 17, 2022, 11:40am

I made curl with -k cause I have problems with certs. But I start Fleet with --fleet-server-es-insecure, so I think it doesn't makes troubles for me.

root@fleet-server:~# curl -k -u ***:*** https://***:9200
{
  "name" : "elasticsearch-master-2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "5F88vkM4QjK2r2t4yUOJkw",
  "version" : {
    "number" : "8.4.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "2bd229c8e56650b42e40992322a76e7914258f0c",
    "build_date" : "2022-08-26T12:11:43.232597118Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Julia_Bardi · November 21, 2022, 10:35am

Can you check the output of this API request? You can find the policy id if you look at the URL of Edit Fleet Server integration page.
https://<KIBANA_HOST:PORT>/api/fleet/package_policies/<id of fleet-server integration policy>

I would like to see if the inputs are generated correctly.

leschev · November 22, 2022, 10:14am

Sure.

{
  "item": {
    "id": "13bcfd99-a40a-4b46-9bd9-395f76a0f0fa",
    "version": "WzEyNjg1MywyXQ==",
    "name": "fleet_server-1",
    "namespace": "default",
    "description": "",
    "package": {
      "name": "fleet_server",
      "title": "Fleet Server",
      "version": "1.2.0"
    },
    "enabled": true,
    "policy_id": "fleet-server-policy",
    "output_id": "",
    "inputs": [
      {
        "type": "fleet-server",
        "policy_template": "fleet_server",
        "enabled": true,
        "streams": [],
        "vars": {
          "host": {
            "type": "text",
            "value": "172.24.54.23"
          },
          "port": {
            "value": [
              8220
            ],
            "type": "integer"
          },
          "max_agents": {
            "type": "integer"
          },
          "max_connections": {
            "type": "integer"
          },
          "custom": {
            "value": "",
            "type": "yaml"
          }
        },
        "compiled_input": {
          "server": {
            "port": 8220,
            "host": "172.24.54.23"
          }
        }
      }
    ],
    "revision": 1,
    "created_at": "2022-11-16T15:58:52.740Z",
    "created_by": "leschev",
    "updated_at": "2022-11-16T15:58:52.740Z",
    "updated_by": "leschev"
  }
}

Julia_Bardi · November 22, 2022, 10:34am

I don't see anything wrong with the policy input.
What happens if you try the enroll without --fleet-server-es-insecure flag?

Also can you show the Settings page on the UI, I am curious is the Elasticsearch Output is set up correctly.

leschev · November 22, 2022, 12:15pm

The same error.

sudo ./elastic-agent install -f  --fleet-server-es=https://****:9200   --fleet-server-service-token=AAEAA****   --fleet-server-policy=fleet-server-policy
{"log.level":"info","@timestamp":"2022-11-22T12:09:52.465Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-22T12:10:03.688Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":447},"message":"Retrying to restart...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-22T12:10:04.692Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: fleet-server-policy","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-22T12:10:06.695Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-22T12:10:10.696Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: fleet-server-policy","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html

Julia_Bardi · November 22, 2022, 4:28pm

Could you also check the result of this ES query, to see if the policy record contains the fleet-server input?

GET .fleet-policies/_search?q=fleet-server-policy

leschev · November 22, 2022, 6:20pm

Could you also check the result of this ES query, to see if the policy record contains the fleet-server input?

pastebin

Also can you show the Settings page on the UI, I am curious is the Elasticsearch Output is set up correctly.

Julia_Bardi · November 23, 2022, 8:03am

I don't see anything wrong in the output. Does the elasticsearch output host look good to you?

One thing you can try is resetting the fleet server policy:

curl --request POST
--url https://{YOUR_KIBANA}/internal/fleet/reset_preconfigured_agent_policies/fleet-server-policy
-u elastic:<your password>
--header 'Content-Type: application/json'
--header 'kbn-xsrf: debug'

leschev · November 23, 2022, 10:51am

All hosts in cluster are green. Nothing problems else.

Cluster we use are configured with SSO so my request is more complex, but I get 404:

curl 'https://{KIBANA}/internal/fleet/reset_preconfigured_agent_policies/fleet-server-policy' \
  -H 'authority: {KIBANA}' \
  -H 'accept: */*' \
  -H 'accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7' \
  -H 'content-type: application/json' \
  -H 'cookie: XXXX' \
  -H 'kbn-version: 8.4.1' \
  -H 'referer: https://{KIBANA}/app/fleet/policies' \
  -H 'sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Linux"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36' \
  -H 'x-kbn-context: %7B%22name%22%3A%22fleet%22%2C%22url%22%3A%22%2Fapp%2Ffleet%22%7D' \
  -H 'kbn-xsrf: debug' --compressed

And response:

{"statusCode":404,"error":"Not Found","message":"Not Found"}

How can I make it via UI? I've tried to remove and create fleet server policy earlier, but it is not helped me.

Julia_Bardi · November 23, 2022, 11:51am

Since 8.4.0 we have a Debug UI in Fleet where you can trigger the Reset of preconfigured policies.
Though I am not sure if your policy is preconfigured, so this might not help.

https://KIBANA/app/fleet/_debug

leschev · November 23, 2022, 12:54pm

Yes. There are not any options available here. I've deleted policy from this debug menu and generated it from quick start. Nothing changed. This is the new generated policy - pastebin

Julia_Bardi · November 23, 2022, 1:19pm

I had a look at the fleet-server logic and the contents of the .fleet-policies index sent earlier. According to the logic, fleet-server takes the latest revision_idx from the policies index, and I see 2 documents with latest revision 4, one of them doesn't contain a fleet-server input. I don't know how this could happen, but there is a workaround if this is the issue.

Can you try creating another Fleet Server policy with a different id, and try to use that to enroll a Fleet server?
You can pass it to the enroll command with this parameter --fleet-server-policy=cutom_id

leschev · November 23, 2022, 1:52pm

What I have did:

removed fleet-server-policy
generated new policy with name Test Fleet Server Policy on this page: https://{KIBANA}/app/fleet/agents on "Advanced" tab
tried to install fleet-server applied --fleet-server-policy=test-fleet-server-policy

And nothing changes :c The same error.

Cristina_Amico · November 23, 2022, 2:36pm

Hi @leschev,

I suspect that there might be a problem in the command you used to install the fleet server. I tried locally using the same command and failed for me with "context canceled". Could you try with this instead?

sudo ./elastic-agent install -f \
  --fleet-server-es=http://localhost:9200 \
  --fleet-server-service-token=**** \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-es-insecure

The difference is that it's exposing the ES connection on http rather than https. Let us know if this works for you.

leschev · November 23, 2022, 2:48pm

Hello!

I've tried to change https to http:

./elastic-agent install -f  --fleet-server-es=http://****:9200   --fleet-server-service-token=****   --fleet-server-policy=test-fleet-server-policy
{"log.level":"info","@timestamp":"2022-11-23T14:45:11.193Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-23T14:45:14.961Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Error - EOF","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html

With --fleet-server-es-insecure the same result.

Cristina_Amico · November 23, 2022, 3:05pm

Could you also change the "Elasticsearch Host" in outputs to http protocol and then re run the command? They need to be the same in order to work properly.

leschev · November 23, 2022, 4:06pm

Sure. I've changed it to http and tried. The same error with EOF.

You can make sure. This is settings page from UI:

Cristina_Amico · November 24, 2022, 8:38am

Hi @leschev,

it's a bit difficult to diagnose your issue without having more info.

Could you share some more details around your configuration (like the versions of the installed stack, how they were installed etc) and also your kibana.yml?