Waiting on default policy with Fleet Server integration

Elastic Stack Elasticsearch
1

Hello,
I am not able to run fleet 8.2.2 on EKS Kubernetes. I am not able to find what I'm missing and need to your adises. Here is my config:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: $FLEETSERVERNAME
  namespace: $NAMESPACE
spec:
  version: 8.1.2
  mode: fleet
  fleetServerEnabled: true
  kibanaRef:
    name: $KIBANANAME
    namespace: $NAMESPACE
  elasticsearchRefs:
  - name: $ESCLUSTERNAME
    namespace: $NAMESPACE
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
          - name: agent
            env:
              - name: FLEET_INSECURE
                value: "true"
              - name: FLEET_ENROLLMENT_TOKEN
                value: $FLEETENROLLMENTTOKEN
              - name: ELASTICSEARCH_HOST
                value: https://$ESCLUSTERNAME-es-http.$NAMESPACE.svc
              - name: ELASTICSEARCH_PORT
                value: "9200"
              - name: ELASTICSEARCH_USERNAME
                value: elastic
              - name: ELASTICSEARCH_PASSWORD
                value: $ESPASSWORD
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
              - name: KIBANA_FLEET_HOST
                value: "https://$KIBANANAME-kb-http.$NAMESPACE.svc:5601"
              - name: KIBANA_FLEET_PASSWORD
                value: $ESPASSWORD
              - name: KIBANA_FLEET_USERNAME
                value: elastic
              - name: FLEET_SERVER_SERVICE_TOKEN
                value: "XXXXXXXXXXXX"
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: $KIBANANAME
  namespace: $NAMESPACE
spec:
  version: 8.1.2
  count: 1
  http:
    tls:
      certificate:
        secretName: $KIBANANAME
  elasticsearchRef:
    name: $ESCLUSTERNAME
    namespace: $NAMESPACE
  config:
    xpack.reporting.roles.enabled: false
    xpack.reporting.kibanaServer.hostname: "localhost"
    xpack.fleet.agents.elasticsearch.hosts: ["https://$ESCLUSTERNAME-es-http.$NAMESPACE.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.$NAMESPACE.svc:8220"]
    xpack.security.encryptionKey: "ZN1bLO6DZkXsGYUtmpvCxmZSBT4jAv5TKQMyTsAiS7oPYEitbiS7ucVfu34uNPZQ"
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: log
        version: latest
      - name: fleet_server
        version: latest
      - name: kubernetes
        version: 0.14.0

    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        is_default_fleet_server: true
        namespace: $NAMESPACE
        monitoring_enabled:
          - logs
          - metrics
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server

      - name: Elastic Agent on ECK policy
        id: eck-agent
        is_default: true
        namespace: $NAMESPACE
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
        - name: system-1
          id: system-1
          package:
            name: system
        - package:
            name: log
          name: log-1
          inputs:
          - type: logfile
            enabled: true
            streams:
            - data_stream:
                dataset: log.log
              enabled: true
              vars:
              - name: paths
                value: |
                  - '/var/log/containers/*\${kubernetes.container.id}.log'
              - name: custom
                value: |
                  symlinks: true
                  condition: \${kubernetes.namespace} == '$NAMESPACE'
        - package:
            name: kubernetes
          name: kubernetes-1
        - package:
            name: system
          name: system-1
  podTemplate:
    spec:
      automountServiceAccountToken: true
      containers:
        - name: kibana
          env:
            - name: KIBANA_FLEET_SETUP
              value: "1"
            - name: KIBANA_FLEET_HOST
              value: "https://$KIBANANAME-kb-http:5601"
            - name: FLEET_INSECURE
              value: "1"
            - name: SERVER_PUBLICBASEURL
              value: "https://$KIBANANAME.u2.ku.edu.tr"
            - name: KIBANA_HOST
              value: "https://$KIBANANAME-kb-http:5601"

...getting following logs:

Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Performing setup of Fleet in Kibana

{"log.level":"info","@timestamp":"2022-04-13T08:33:39.032Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":572},"message":"Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:39.436Z","log.origin":{"file.name":"application/application.go","file.line":68},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:39.437Z","log.origin":{"file.name":"application/application.go","file.line":88},"message":"Agent is in Fleet Server bootstrap mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:40.121Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":744},"message":"Waiting for Elastic Agent to start Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:40.333Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":62},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:40.333Z","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":131},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:40.334Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":64},"message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:40.420Z","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":141},"message":"Agent is stopped","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:40.423Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is 5ju45wyJ","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:40.423Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 1 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:42.533Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-04-13T08:33:42Z - message: Application: fleet-server--8.1.2[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:42.534Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:43.729Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-04-13T08:33:43Z - message: Application: fleet-server--8.1.2[]: State changed to STARTING: Waiting on default policy with Fleet Server integration - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-13T08:33:46.126Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":777},"message":"Fleet Server - Waiting on default policy with Fleet Server integration","ecs.version":"1.6.0"}

Fleet cannot register itself. I tried quickstart yaml files; same result.

Thank & Regards

2

Hi, could you check Kibana logs to see if there were any errors when creating agent policies on startup?

3

I found the problem, I added FLEET_SERVER_POLICY_ID and the problem is evolved to a different level.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.