Fleet Server not deploying in ECK operator

This is my fleet.yaml file , i cant seem to understand why im getting this error

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  annotations:
    association.k8s.elastic.co/es-conf-3796208044: >-
      {"authSecretName":"fleet-server-default-logging-elasticsearch-default-agent-user","authSecretKey":"token","isServiceAccount":true,"caCertProvided":true,"caSecretName":"fleet-server-agent-es-lmco-logging-elasticsearch-default-ca","url":"https://elasticsearch-default-es-http.default-logging.svc:9200","version":"8.4.0"}
    association.k8s.elastic.co/kb-conf: >-
      {"authSecretName":"fleet-server-agent-kb-user","authSecretKey":"default-logging-fleet-server-agent-kb-user","isServiceAccount":false,"caCertProvided":true,"caSecretName":"fleet-server-agent-kibana-ca","url":"https://kibana-default-kb-http.lmco-logging.svc:5601","version":"8.4.0"}
    fleet.eck.k8s.elastic.co/token: c79cbc9e-f251-4fe8-965d-0e1b4520c82c
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"agent.k8s.elastic.co/v1alpha1","kind":"Agent","metadata":{"annotations":{},"name":"fleet-server","namespace":"default-logging"},"spec":{"deployment":{"podTemplate":{"spec":{"automountServiceAccountToken":true,"serviceAccountName":"fleet-server"}},"replicas":1},"elasticsearchRefs":[{"name":"elasticsearch-lmco","namespace":"default-logging","serviceName":"elasticsearch-default-es-default"}],"fleetServerEnabled":true,"kibanaRef":{"name":"kibana-default"},"mode":"fleet","version":"8.4.0"}}
  creationTimestamp: '2023-06-22T20:17:14Z'
  generation: 14
  managedFields:
    - apiVersion: agent.k8s.elastic.co/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
        'f:spec':
          .: {}
          'f:deployment':
            .: {}
            'f:podTemplate':
              .: {}
              'f:spec':
                .: {}
                'f:automountServiceAccountToken': {}
            'f:replicas': {}
          'f:fleetServerEnabled': {}
          'f:kibanaRef':
            .: {}
            'f:name': {}
          'f:mode': {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2023-06-22T20:17:14Z'
    - apiVersion: agent.k8s.elastic.co/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:association.k8s.elastic.co/es-conf-3796208044': {}
            'f:association.k8s.elastic.co/kb-conf': {}
            'f:fleet.eck.k8s.elastic.co/token': {}
        'f:spec':
          'f:deployment':
            'f:podTemplate':
              'f:metadata':
                .: {}
                'f:creationTimestamp': {}
              'f:spec':
                'f:containers': {}
            'f:strategy': {}
          'f:fleetServerRef': {}
          'f:http':
            .: {}
            'f:service':
              .: {}
              'f:metadata': {}
              'f:spec': {}
            'f:tls':
              .: {}
              'f:certificate': {}
      manager: elastic-operator
      operation: Update
      time: '2023-06-27T19:50:55Z'
    - apiVersion: agent.k8s.elastic.co/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          .: {}
          'f:availableNodes': {}
          'f:elasticsearchAssociationsStatus':
            .: {}
            'f:lmco-logging/elasticsearch-lmco': {}
          'f:expectedNodes': {}
          'f:health': {}
          'f:kibanaAssociationStatus': {}
          'f:observedGeneration': {}
          'f:version': {}
      manager: elastic-operator
      operation: Update
      subresource: status
      time: '2023-06-28T15:00:54Z'
    - apiVersion: agent.k8s.elastic.co/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:deployment':
            'f:podTemplate':
              'f:spec':
                'f:securityContext':
                  .: {}
                  'f:runAsUser': {}
                'f:serviceAccountName': {}
          'f:elasticsearchRefs': {}
          'f:policyID': {}
          'f:version': {}
      manager: Mozilla
      operation: Update
      time: '2023-06-28T15:01:49Z'
  name: fleet-server
  namespace: defaulttlogging
  resourceVersion: '155344211'
  uid: 807c4b29-4fc8-4729-a940-410c5127d2dd
spec:
  fleetServerRef: {}
  mode: fleet
  kibanaRef:
    name: kibana-default
  deployment:
    podTemplate:
      metadata:
        creationTimestamp: null
      spec:
        automountServiceAccountToken: true
        containers:
          - name: agent
            resources: {}
            securityContext:
              privileged: true
              runAsUser: 0
        securityContext:
          runAsUser: 0
        serviceAccountName: fleet-server
    replicas: 1
    strategy: {}
  http:
    service:
      metadata: {}
      spec: {}
    tls:
      certificate: {}
  policyID: eck-fleet-server
  fleetServerEnabled: true
  version: 8.7.0
  elasticsearchRefs:
    - name: elasticsearch-default
      namespace: default
status:
  availableNodes: 1
  elasticsearchAssociationsStatus:
    default-logging/elasticsearch-default: Established
  expectedNodes: 1
  health: green
  kibanaAssociationStatus: Established
  observedGeneration: 14
  version: 8.4.0

Error

{"log.level":"info","@timestamp":"2023-06-28T16:15:28.484Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":587},"message":"Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:28.884Z","log.origin":{"file.name":"cmd/run.go","file.line":202},"message":"APM instrumentation disabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:28.886Z","log.origin":{"file.name":"application/application.go","file.line":78},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:28.962Z","log.origin":{"file.name":"application/application.go","file.line":98},"message":"Agent is in Fleet Server bootstrap mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:29.556Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":759},"message":"Waiting for Elastic Agent to start Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:29.763Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":68},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:29.764Z","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":134},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:29.764Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":70},"message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:29.765Z","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":144},"message":"Agent is stopped","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:29.767Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is k4ZK5yVd","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:29.767Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 1 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:29.965Z","log.origin":{"file.name":"operation/operator.go","file.line":348},"message":"operation 'operation-install' skipped for fleet-server.8.4.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:31.554Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2023-06-28T16:15:31Z - message: Application: fleet-server--8.4.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:31.556Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:31.557Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-06-28T16:15:31.664Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"fleet-server stderr: \"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2023-06-28T16:15:31Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2023-06-28T16:15:31Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n\"","agent.console.name":"fleet-server","agent.console.type":"stderr","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:32.761Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2023-06-28T16:15:32Z - message: Application: fleet-server--8.4.0[]: State changed to STARTING: Waiting on fleet-server input to be added to policy: eck-fleet-server - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:15:35.560Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: eck-fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:17:28.486Z","log.origin":{"file.name":"cmd/run.go","file.line":254},"message":"Shutting down Elastic Agent and sending last events...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:17:28.486Z","log.origin":{"file.name":"operation/operator.go","file.line":263},"message":"waiting for installer of pipeline 'default' to finish","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:17:28.486Z","log.origin":{"file.name":"process/app.go","file.line":182},"message":"Signaling application to stop because of shutdown: fleet-server--8.4.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-06-28T16:17:28.987Z","log.origin":{"file.name":"process/app.go","file.line":290},"message":"failed to stop fleet-server: os: process already finished","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:17:29.987Z","log.origin":{"file.name":"cmd/run.go","file.line":262},"message":"Shutting down completed.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:17:29.987Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2023-06-28T16:17:29Z - message: Application: fleet-server--8.4.0[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-28T16:17:29.987Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":72},"message":"Stats endpoint (/usr/share/elastic-agent/state/data/tmp/elastic-agent.sock) finished: accept unix /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock: use of closed network connection","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
Error: enrollment failed: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html

Hi Chiderah, could you share the content of the eck-fleet-server policy? From the error message it seems that the fleet-server input is not correctly configured. You need a config similar to this: https://github.com/elastic/cloud-on-k8s/blob/main/config/recipes/elastic-agent/fleet-kubernetes-integration.yaml#L34

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.