Hi there,
I cannot get my fleet server to communicate with Kibana. I have followed extensive documentation and examples and I can't get anything to work.
Here is the yaml files
fleet-server.yaml
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: fleet-server
namespace: elastic-system
spec:
version: 8.14.3
kibanaRef:
name: kibana
elasticsearchRefs:
- name: elasticsearch
mode: fleet
fleetServerEnabled: true
policyID: eck-fleet-server
deployment:
replicas: 1
podTemplate:
spec:
serviceAccountName: elastic-agent
automountServiceAccountToken: true
securityContext:
runAsUser: 0
http:
service:
spec:
type: LoadBalancer
the logs from Kibana
2024-07-28T07:24:13.147+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.348+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.456+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.565+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.679+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.786+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:13.893+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:14.053+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:14.799+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic-system-fleet-server-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2024-07-28T07:24:16.203+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2024-07-28T07:24:16.203+00:00][INFO ][plugins.fleet] Cleaning old indices
[2024-07-28T07:24:16.424+00:00][INFO ][plugins.fleet] Fleet setup completed
[2024-07-28T07:28:03.837+00:00][INFO ][plugins.security.authentication] Performing login attempt with "basic" provider.
[2024-07-28T07:28:05.303+00:00][INFO ][plugins.security.authentication] Login attempt with "basic" provider succeeded (requires redirect: true).
[2024-07-28T07:28:16.548+00:00][INFO ][plugins.fleet] Secrets storage is disabled as minimum fleet server version has not been me
the logs from events
elastic-system 9m45s Warning AssociationError agent/fleet-server Association backend for kibana is not configured
elastic-system 9m45s Normal AssociationStatusChange agent/fleet-server Association status changed from [] to [Established]
elastic-system 9m45s Normal AssociationStatusChange agent/fleet-server Association status changed from [] to [elastic-system/elasticsearch: Established]
elastic-system 9m45s Normal IPAllocated service/fleet-server-agent-http Assigned IP ["192.168.0.178"]
elastic-system 9m45s Warning AssociationError agent/fleet-server Association backend for elasticsearch is not configured
elastic-system 9m33s Warning ReconciliationError agent/fleet-server Reconciliation error: failed to request https://kibana-kb-http.elastic-system.svc:5601/api/fleet/setup, status is 401)
elastic-system 8m21s Warning FailedCreate replicaset/fleet-server-agent-7488458fd8 Error creating: pods "fleet-server-agent-7488458fd8-86s2p" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system 8m21s Warning FailedCreate replicaset/fleet-server-agent-7488458fd8 Error creating: pods "fleet-server-agent-7488458fd8-hdc2z" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system 8m21s Warning FailedCreate replicaset/fleet-server-agent-7488458fd8 Error creating: pods "fleet-server-agent-7488458fd8-qv78c" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system 8m21s Warning FailedCreate replicaset/fleet-server-agent-7488458fd8 Error creating: pods "fleet-server-agent-7488458fd8-vws4z" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system 8m21s Warning FailedCreate replicaset/fleet-server-agent-7488458fd8 Error creating: pods "fleet-server-agent-7488458fd8-2t97r" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system 8m21s Warning FailedCreate replicaset/fleet-server-agent-7488458fd8 Error creating: pods "fleet-server-agent-7488458fd8-t5jlk" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system 8m21s Warning FailedCreate replicaset/fleet-server-agent-7488458fd8 Error creating: pods "fleet-server-agent-7488458fd8-8tw9h" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system 8m21s Normal ScalingReplicaSet deployment/fleet-server-agent Scaled up replica set fleet-server-agent-7488458fd8 to 1
elastic-system 8m20s Warning FailedCreate replicaset/fleet-server-agent-7488458fd8 Error creating: pods "fleet-server-agent-7488458fd8-8cb4g" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system 8m20s Warning FailedCreate replicaset/fleet-server-agent-7488458fd8 Error creating: pods "fleet-server-agent-7488458fd8-vg6cn" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
elastic-system 2m53s Warning FailedCreate replicaset/fleet-server-agent-7488458fd8 (combined from similar events): Error creating: pods "fleet-server-agent-7488458fd8-8545p" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "agent-data")
My understanding is the Warning when that actually make it not deploy but the line where it says it's failing to request the url to get everything it needs I don't understand what I need to do to fix this .
I've been working on this for months in my lab and I feel like it real close the Kubernetes integration wasn't working via the template so I moved to using the agents via a different template that it was taken from the gui of the cabana server. This is all just random information but fleet worked before but the flea information in the Yaml haven't changed? Any help anyone could provide or any information that I can give I would be very very very very grateful
I mean I've been posting everywhere i'm not above paying someone 75 $ for a hour troubleshooting session