I'm trying to deploy ECK on a local kubernetes cluster with below given spec.
I'm trying to serve kibana ingress using /beats, hence I changed/updated those parameters in config section.
the fleet server never comes online and I can't find out the way out.
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana
spec:
version: 8.12.1
count: 1
podTemplate:
spec:
containers:
- name: kibana
livenessProbe:
tcpSocket:
port: 5601
initialDelaySeconds: 5
periodSeconds: 20
timeoutSeconds: 10
successThreshold: 1
failureThreshold: 30
readinessProbe:
tcpSocket:
port: 5601
initialDelaySeconds: 5
periodSeconds: 20
timeoutSeconds: 10
successThreshold: 1
failureThreshold: 30
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=2048"
resources:
requests:
memory: 2Gi
cpu: 0.5
limits:
memory: 2Gi
cpu: 1
elasticsearchRef:
name: elasticsearch
config:
server.basePath: /beats
server.rewriteBasePath: true
xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.default.svc:9200"]
xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.default.svc:8220"]
xpack.fleet.packages:
- name: system
version: latest
- name: elastic_agent
version: latest
- name: fleet_server
version: latest
- name: kubernetes
version: latest
xpack.fleet.agentPolicies:
- name: Fleet Server on ECK policy
id: eck-fleet-server
namespace: default
monitoring_enabled:
- logs
- metrics
unenroll_timeout: 900
package_policies:
- name: fleet_server-1
id: fleet_server-1
package:
name: fleet_server
- name: Elastic Agent on ECK policy
id: eck-agent
namespace: default
monitoring_enabled:
- logs
- metrics
unenroll_timeout: 900
package_policies:
- package:
name: system
name: system-1
- package:
name: kubernetes
name: kubernetes-1
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch
spec:
version: 8.12.1
nodeSets:
- name: default
count: 3
config:
node.store.allow_mmap: true
podTemplate:
spec:
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: -Xms2g -Xmx2g
resources:
requests:
memory: 2Gi
cpu: 1
limits:
cpu: 1
memory: 4Gi
volumeClaimTemplates:
- metadata:
name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
# storageClassName: standard
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: fleet-server
spec:
version: 8.12.1
kibanaRef:
name: kibana
elasticsearchRefs:
- name: elasticsearch
mode: fleet
fleetServerEnabled: true
policyID: eck-fleet-server
deployment:
replicas: 1
podTemplate:
spec:
serviceAccountName: fleet-server
automountServiceAccountToken: true
securityContext:
runAsUser: 0
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: elastic-agent
spec:
version: 8.12.1
kibanaRef:
name: kibana
fleetServerRef:
name: fleet-server
mode: fleet
policyID: eck-agent
daemonSet:
podTemplate:
spec:
serviceAccountName: elastic-agent
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
automountServiceAccountToken: true
securityContext:
runAsUser: 0
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fleet-server
rules:
- apiGroups: [""]
resources:
- pods
- namespaces
- nodes
verbs:
- get
- watch
- list
- apiGroups: ["apps"]
resources:
- replicasets
verbs:
- get
- watch
- list
- apiGroups: ["batch"]
resources:
- jobs
verbs:
- get
- watch
- list
- apiGroups: ["coordination.k8s.io"]
resources:
- leases
verbs:
- get
- create
- update
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fleet-server
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: fleet-server
subjects:
- kind: ServiceAccount
name: fleet-server
namespace: default
roleRef:
kind: ClusterRole
name: fleet-server
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: elastic-agent
rules:
- apiGroups: [""]
resources:
- pods
- nodes
- namespaces
- events
- services
- configmaps
verbs:
- get
- watch
- list
- apiGroups: ["coordination.k8s.io"]
resources:
- leases
verbs:
- get
- create
- update
- nonResourceURLs:
- "/metrics"
verbs:
- get
- apiGroups: ["extensions"]
resources:
- replicasets
verbs:
- "get"
- "list"
- "watch"
- apiGroups:
- "apps"
resources:
- statefulsets
- deployments
- replicasets
verbs:
- "get"
- "list"
- "watch"
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
- apiGroups:
- "batch"
resources:
- jobs
verbs:
- "get"
- "list"
- "watch"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: elastic-agent
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: elastic-agent
subjects:
- kind: ServiceAccount
name: elastic-agent
namespace: default
roleRef:
kind: ClusterRole
name: elastic-agent
apiGroup: rbac.authorization.k8s.io
Now everything comes online except fleet server which is complaining about Kibana endpoint not accessible
k describe agent fleet-server
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AssociationStatusChange 22m agent-es-association-controller Association status changed from [] to [default/elasticsearch: Pending]
Normal AssociationStatusChange 22m agent-kibana-association-controller Association status changed from [] to [Pending]
Warning AssociationError 22m (x7 over 22m) agent-controller Association backend for elasticsearch is not configured
Warning AssociationError 22m (x12 over 22m) agent-controller Association backend for kibana is not configured
Normal AssociationStatusChange 22m agent-es-association-controller Association status changed from [default/elasticsearch: Pending] to [default/elasticsearch: Established]
Normal AssociationStatusChange 22m agent-kibana-association-controller Association status changed from [Pending] to [Established]
Warning Delayed 22m (x5 over 22m) agent-controller Delaying deployment of Elastic Agent in Fleet Mode as Kibana is not available yet
Warning ReconciliationError 22m agent-controller Reconciliation error: secrets "fleet-server-agent-http-certs-public" already exists
Warning ReconciliationError 47s (x6 over 21m) agent-controller Reconciliation error: failed to request https://kibana-kb-http.default.svc:5601/api/fleet/setup, status is 404)