ECK Selfhosted k8's fleet -server wont connect Kibana

Hi there,

I cannot get my fleet server to communicate with Kibana. I have followed extensive documentation and examples and I can't get anything to work.
Here is the yaml files

fleet-server.yaml
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
  namespace: elastic-system
spec:
  version: 8.14.3
  kibanaRef:
    name: kibana
  elasticsearchRefs:
  - name: elasticsearch
  mode: fleet
  fleetServerEnabled: true
  policyID: eck-fleet-server
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
  http:
    service:
      spec:
        type: LoadBalancer

the logs from Kibana

2024-07-28T07:24:13.147+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.348+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.456+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.565+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.679+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.786+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.893+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:14.053+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:14.799+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:16.203+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2024-07-28T07:24:16.203+00:00][INFO ][plugins.fleet] Cleaning old indices
[2024-07-28T07:24:16.424+00:00][INFO ][plugins.fleet] Fleet setup completed
[2024-07-28T07:28:03.837+00:00][INFO ][plugins.security.authentication] Performing login attempt with "basic" provider.
[2024-07-28T07:28:05.303+00:00][INFO ][plugins.security.authentication] Login attempt with "basic" provider succeeded (requires redirect: true).
[2024-07-28T07:28:16.548+00:00][INFO ][plugins.fleet] Secrets storage is disabled as minimum fleet server version has not been me

the logs from events

elastic-system   9m45s       Warning   AssociationError          agent/fleet-server                         Association backend for kibana is not configured
elastic-system   9m45s       Normal    AssociationStatusChange   agent/fleet-server                         Association status changed from [] to [Established]
elastic-system   9m45s       Normal    AssociationStatusChange   agent/fleet-server                         Association status changed from [] to [elastic-system/elasticsearch: Established]
elastic-system   9m45s       Normal    IPAllocated               service/fleet-server-agent-http            Assigned IP ["192.168.0.178"]
elastic-system   9m45s       Warning   AssociationError          agent/fleet-server                         Association backend for elasticsearch is not configured
elastic-system   9m33s       Warning   ReconciliationError       agent/fleet-server                         Reconciliation error: failed to request https://kibana-kb-http.elastic-system.svc:5601/api/fleet/setup, status is 401)
elastic-system   8m21s       Warning   FailedCreate              replicaset/fleet-server-agent-7488458fd8   Error creating: pods "fleet-server-agent-7488458fd8-86s2p" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system   8m21s       Warning   FailedCreate              replicaset/fleet-server-agent-7488458fd8   Error creating: pods "fleet-server-agent-7488458fd8-hdc2z" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system   8m21s       Warning   FailedCreate              replicaset/fleet-server-agent-7488458fd8   Error creating: pods "fleet-server-agent-7488458fd8-qv78c" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system   8m21s       Warning   FailedCreate              replicaset/fleet-server-agent-7488458fd8   Error creating: pods "fleet-server-agent-7488458fd8-vws4z" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system   8m21s       Warning   FailedCreate              replicaset/fleet-server-agent-7488458fd8   Error creating: pods "fleet-server-agent-7488458fd8-2t97r" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system   8m21s       Warning   FailedCreate              replicaset/fleet-server-agent-7488458fd8   Error creating: pods "fleet-server-agent-7488458fd8-t5jlk" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system   8m21s       Warning   FailedCreate              replicaset/fleet-server-agent-7488458fd8   Error creating: pods "fleet-server-agent-7488458fd8-8tw9h" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system   8m21s       Normal    ScalingReplicaSet         deployment/fleet-server-agent              Scaled up replica set fleet-server-agent-7488458fd8 to 1
elastic-system   8m20s       Warning   FailedCreate              replicaset/fleet-server-agent-7488458fd8   Error creating: pods "fleet-server-agent-7488458fd8-8cb4g" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system   8m20s       Warning   FailedCreate              replicaset/fleet-server-agent-7488458fd8   Error creating: pods "fleet-server-agent-7488458fd8-vg6cn" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system   2m53s       Warning   FailedCreate              replicaset/fleet-server-agent-7488458fd8   (combined from similar events): Error creating: pods "fleet-server-agent-7488458fd8-8545p" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")

My understanding is the Warning when that actually make it not deploy but the line where it says it's failing to request the url to get everything it needs I don't understand what I need to do to fix this .

I've been working on this for months in my lab and I feel like it real close the Kubernetes integration wasn't working via the template so I moved to using the agents via a different template that it was taken from the gui of the cabana server. This is all just random information but fleet worked before but the flea information in the Yaml haven't changed? Any help anyone could provide or any information that I can give I would be very very very very grateful
I mean I've been posting everywhere i'm not above paying someone 75 $ for a hour troubleshooting session

this did it
kubectl label ns pod-security.kubernetes.io/enforce=privileged