Error: fail to enroll: fail to execute request to fleet-server: status code: 400, fleet-server returned an error: BadRequest

[tirelibirefe (Tireli Efe)]

Hello,
I am not able to run Fleet on Kubernetes.

Here is my config:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: $FLEETSERVERNAME
  namespace: $NAMESPACE
spec:
  version: 8.1.2
  mode: fleet
  fleetServerEnabled: true
  kibanaRef:
    name: $KIBANANAME
    namespace: $NAMESPACE
  elasticsearchRefs:
  - name: $ESCLUSTERNAME
    namespace: $NAMESPACE
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
          - name: agent
            volumeMounts:
            - mountPath: /mnt/elastic-internal/kibana-association/logging/kibana/certs
              name: kibana-kb-es-ca
              readOnly: true
            - mountPath: /mnt/elastic-internal/fleetserver-association/logging/fleet-server/certs
              name: fleet-server-agent-http-certs-internal
              readOnly: true
            env:
              - name: FLEET_ENROLLMENT_TOKEN
                value: $FLEETENROLLMENTTOKEN
              - name: ELASTICSEARCH_HOST
                value: https://$ESCLUSTERNAME-es-http.$NAMESPACE.svc
              - name: ELASTICSEARCH_PORT
                value: "9200"
              - name: ELASTICSEARCH_USERNAME
                value: elastic
              - name: ELASTICSEARCH_PASSWORD
                value: $ESPASSWORD
              - name: FLEET_SERVER_SERVICE_TOKEN
                value: $FLEETSERVERSERVICETOKEN
              - name: FLEET_SERVER_POLICY_ID
                value: "eck-fleet-server"
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
        volumes:
        - name: kibana-kb-es-ca
          secret:
            secretName: kibana-monitoring
        - name: fleet-server-agent-http-certs-internal
          secret:
            secretName: fleet-server-agent-http-certs-internal

...and the logs:

{"log.level":"info","@timestamp":"2022-04-14T08:20:15.789Z","log.origin":{"file.name":"cmd/run.go","file.line":185},"message":"Shutting down Elastic Agent and sending last events...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T08:20:15.789Z","log.origin":{"file.name":"operation/operator.go","file.line":216},"message":"waiting for installer of pipeline 'default' to finish","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T08:20:15.790Z","log.origin":{"file.name":"process/app.go","file.line":176},"message":"Signaling application to stop because of shutdown: fleet-server--8.1.2","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T08:20:17.291Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T08:20:17.291Z","log.origin":{"file.name":"cmd/run.go","file.line":193},"message":"Shutting down completed.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T08:20:17.292Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":66},"message":"Stats endpoint (/usr/share/elastic-agent/state/data/tmp/elastic-agent.sock) finished: accept unix /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock: use of closed network connection","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: status code: 400, fleet-server returned an error: BadRequest
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.1/fleet-troubleshooting.html
Error: enrollment failed: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.1/fleet-troubleshooting.html

Could you please advise which request supposed as bad by Fleet?

Thanks & Regards

[pebrc (Peter Brachwitz)]

What I don't understand about your configuration is why are you recreating all these internal volume mounts and Elasticsearch output configuration environment variables? And where is the FLEET_ENROLLMENT_TOKEN coming from, have you created that ahead of time? I suspect this is causing the error if the token does not exist or is invalid?

The idea behind elasticsearchRef and kibanaRef is that you don't have to worry about all these configuration settings and the ECK operator manages those instead for you.

So a minimal Fleet Server Agent spec looks like this (taken directly from our Quickstart)

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-quickstart
  namespace: default
spec:
  version: 8.1.2
  kibanaRef:
    name: kibana-quickstart
  elasticsearchRefs:
  - name: elasticsearch-quickstart
  mode: fleet
  fleetServerEnabled: true
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0

Have a look at Quickstart | Elastic Cloud on Kubernetes [master] | Elastic for the full example.

You then also need an Agents and Elasticsearch and Kibana which is all in the Quickstart.