Fleet server in Kubernetes

Dmitriy_Esin · October 14, 2021, 1:21pm

Hi all!
I noticed that Elastic Agent has variables for containerized deployment if I run this command:
./elastic-agent container -h
Can I deploy Fleet Server, for example in the Kubernetes cluster?
I've tried to use these variables with this yaml :

env:
            - name: FLEET_ENROLL
              value: "1"
            - name: FLEET_INSECURE
              value: "1"
            - name: FLEET_URL
              value: "http://URL:8220"
            - name: FLEET_SERVER_SERVICE_TOKEN
              value: "VALUE"
            - name: FLEET_SERVER_POLICY
              value: "856f4eb0-122f-11ec-8308-176e4cb272f3"
            - name: ELASTICSEARCH_HOST
              value: "https://elasticsearch.em-infra.local:9200"
            - name: KIBANA_FLEET_USERNAME
              value: "USERNAME"
            - name: KIBANA_FLEET_PASSWORD
              value: "PASSWORD"
            - name: FLEET_SERVER_ENABLE
              value: "1"
            - name: FLEET_SERVER_INSECURE_HTTP
              value: "0"
            - name: FLEET_SERVER_ELASTICSEARCH 
              valueFrom:
                secretKeyRef:
                  name: elastic-certificate-pem
                  key: "elastic-certificate.pem"
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

and got this error

Policy selected for enrollment:  MY_POLICY
2021-10-14T13:16:03.420Z	INFO	cmd/enroll_cmd.go:354	Generating self-signed certificate for Fleet Server
2021-10-14T13:16:03.898Z	INFO	cmd/enroll_cmd.go:526	Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-10-14T13:16:04.085Z	INFO	application/application.go:66	Detecting execution mode
2021-10-14T13:16:04.086Z	INFO	application/application.go:87	Agent is in Fleet Server bootstrap mode
2021-10-14T13:16:04.224Z	INFO	[api]	api/server.go:62	Starting stats endpoint
2021-10-14T13:16:04.224Z	INFO	[api]	api/server.go:64	Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)
2021-10-14T13:16:04.224Z	INFO	application/fleet_server_bootstrap.go:124	Agent is starting
2021-10-14T13:16:04.226Z	INFO	application/fleet_server_bootstrap.go:134	Agent is stopped
2021-10-14T13:16:04.228Z	INFO	stateresolver/stateresolver.go:48	New State ID is sxYbVCI3
2021-10-14T13:16:04.228Z	INFO	stateresolver/stateresolver.go:49	Converging state requires execution of 1 step(s)
2021-10-14T13:16:05.174Z	INFO	log/reporter.go:40	2021-10-14T13:16:05Z - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-10-14T13:16:05.177Z	INFO	stateresolver/stateresolver.go:66	Updating internal state
2021-10-14T13:16:05.924Z	INFO	cmd/enroll_cmd.go:701	Fleet Server - Starting
2021-10-14T13:16:06.211Z	ERROR	status/reporter.go:236	Elastic Agent status changed to: 'error'
2021-10-14T13:16:06.212Z	ERROR	log/reporter.go:36	2021-10-14T13:16:06Z - message: Application: fleet-server--7.14.0[]: State changed to FAILED: Error - x509: certificate signed by unknown authority - type: 'ERROR' - sub_type: 'FAILED'
2021-10-14T13:16:06.926Z	INFO	cmd/enroll_cmd.go:701	Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-14T13:16:12.936Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-14T13:16:16.213Z	INFO	status/reporter.go:236	Elastic Agent status changed to: 'online'
2021-10-14T13:16:16.213Z	INFO	log/reporter.go:40	2021-10-14T13:16:16Z - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING:  - type: 'STATE' - sub_type: 'STARTING'
2021-10-14T13:16:16.213Z	INFO	log/reporter.go:40	2021-10-14T13:16:16Z - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-10-14T13:16:16.213Z	INFO	log/reporter.go:40	2021-10-14T13:16:16Z - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-10-14T13:16:16.748Z	INFO	log/reporter.go:40	2021-10-14T13:16:16Z - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-10-14T13:16:16.944Z	INFO	cmd/enroll_cmd.go:701	Fleet Server - Starting
2021-10-14T13:16:17.248Z	ERROR	status/reporter.go:236	Elastic Agent status changed to: 'error'

Maybe anybody has experience with setting up the Elastic Agent with Fleet Server in Kubernetes?

Thank you all!

blaker · October 19, 2021, 3:19pm

Yes we provide documentation on running Elastic Agent in Kubernetes with and without Fleet Server.

Dmitriy_Esin · October 25, 2021, 9:20am

@blaker Hi!
I tried to use this manual today:

Tried to run Fleet Server in k8s container using these env variables

      containers:
        - name: elastic-agent
          image: docker.elastic.co/beats/elastic-agent:7.14.0
          env:
            - name: FLEET_SERVER_ENABLE
              value: "true"
            - name: FLEET_SERVER_ELASTICSEARCH_HOST
              value: "https://XXXXXXXXXXXXXXXX:9200"
            - name: FLEET_SERVER_SERVICE_TOKEN
              value: "XXXXXXXXXXXXXXXXXXXXXXXX"
            - name: KIBANA_FLEET_USERNAME
              value: "elastic"
            - name: KIBANA_FLEET_PASSWORD
              value: "XXXXXXXXXXXXXXXXXXXXXXXX"
            - name: FLEET_SERVER_INSECURE_HTTP
              value: "true"

and I got this error Error - x509: certificate signed by unknown authority in any case (with insecure option or without it)

Policy selected for enrollment:  
2021-10-25T09:11:42.691Z	INFO	cmd/enroll_cmd.go:526	Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-10-25T09:11:42.870Z	INFO	application/application.go:66	Detecting execution mode
2021-10-25T09:11:42.875Z	INFO	application/application.go:87	Agent is in Fleet Server bootstrap mode
2021-10-25T09:11:43.263Z	INFO	[api]	api/server.go:62	Starting stats endpoint
2021-10-25T09:11:43.263Z	INFO	application/fleet_server_bootstrap.go:124	Agent is starting
2021-10-25T09:11:43.264Z	INFO	[api]	api/server.go:64	Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)
2021-10-25T09:11:43.265Z	INFO	application/fleet_server_bootstrap.go:134	Agent is stopped
2021-10-25T09:11:43.267Z	INFO	stateresolver/stateresolver.go:48	New State ID is 6HaGEk3B
2021-10-25T09:11:43.267Z	INFO	stateresolver/stateresolver.go:49	Converging state requires execution of 1 step(s)
2021-10-25T09:11:43.918Z	INFO	log/reporter.go:40	2021-10-25T09:11:43Z - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-10-25T09:11:43.919Z	INFO	stateresolver/stateresolver.go:66	Updating internal state
2021-10-25T09:11:44.696Z	INFO	cmd/enroll_cmd.go:701	Fleet Server - Starting
2021-10-25T09:11:44.954Z	ERROR	status/reporter.go:236	Elastic Agent status changed to: 'error'
2021-10-25T09:11:44.955Z	ERROR	log/reporter.go:36	2021-10-25T09:11:44Z - message: Application: fleet-server--7.14.0[]: State changed to FAILED: Error - x509: certificate signed by unknown authority - type: 'ERROR' - sub_type: 'FAILED'
2021-10-25T09:11:45.698Z	INFO	cmd/enroll_cmd.go:701	Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-25T09:11:51.707Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-25T09:11:54.956Z	INFO	status/reporter.go:236	Elastic Agent status changed to: 'online'
2021-10-25T09:11:54.956Z	INFO	log/reporter.go:40	2021-10-25T09:11:54Z - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING:  - type: 'STATE' - sub_type: 'STARTING'
2021-10-25T09:11:54.957Z	INFO	log/reporter.go:40	2021-10-25T09:11:54Z - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-10-25T09:11:54.957Z	INFO	log/reporter.go:40	2021-10-25T09:11:54Z - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-10-25T09:11:55.491Z	INFO	log/reporter.go:40	2021-10-25T09:11:55Z - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-10-25T09:11:55.719Z	INFO	cmd/enroll_cmd.go:701	Fleet Server - Starting
2021-10-25T09:11:55.992Z	ERROR	status/reporter.go:236	Elastic Agent status changed to: 'error'
2021-10-25T09:11:55.992Z	ERROR	log/reporter.go:36	2021-10-25T09:11:55Z - message: Application: fleet-server--7.14.0[]: State changed to FAILED: Error - x509: certificate signed by unknown authority - type: 'ERROR' - sub_type: 'FAILED'
2021-10-25T09:11:56.721Z	INFO	cmd/enroll_cmd.go:701	Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-25T09:12:02.733Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-25T09:12:05.993Z	INFO	status/reporter.go:236	Elastic Agent status changed to: 'online'

It will be great if you have any advice or solution!
I am stuck on this task.

Dmitriy_Esin · October 25, 2021, 9:32am

When I set "FLEET_SERVER_ELASTICSEARCH_CA" variable I got this error:

Policy selected for enrollment:  
2021-10-25T09:30:06.364Z	INFO	cmd/enroll_cmd.go:526	Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-10-25T09:30:06.533Z	INFO	application/application.go:66	Detecting execution mode
2021-10-25T09:30:06.534Z	INFO	application/application.go:87	Agent is in Fleet Server bootstrap mode
2021-10-25T09:30:06.823Z	INFO	[api]	api/server.go:62	Starting stats endpoint
2021-10-25T09:30:06.823Z	INFO	application/fleet_server_bootstrap.go:124	Agent is starting
2021-10-25T09:30:06.823Z	INFO	[api]	api/server.go:64	Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)
2021-10-25T09:30:06.824Z	INFO	application/fleet_server_bootstrap.go:134	Agent is stopped
2021-10-25T09:30:06.826Z	INFO	stateresolver/stateresolver.go:48	New State ID is mvi4tME1
2021-10-25T09:30:06.827Z	INFO	stateresolver/stateresolver.go:49	Converging state requires execution of 1 step(s)
2021-10-25T09:30:07.605Z	INFO	log/reporter.go:40	2021-10-25T09:30:07Z - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-10-25T09:30:07.607Z	INFO	stateresolver/stateresolver.go:66	Updating internal state
2021-10-25T09:30:08.151Z	INFO	log/reporter.go:40	2021-10-25T09:30:08Z - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING: exited with code: 1 - type: 'STATE' - sub_type: 'STARTING'
2021-10-25T09:30:08.151Z	INFO	log/reporter.go:40	2021-10-25T09:30:08Z - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-10-25T09:30:08.152Z	INFO	log/reporter.go:40	2021-10-25T09:30:08Z - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-10-25T09:30:08.370Z	INFO	cmd/enroll_cmd.go:701	Fleet Server - Restarting
2021-10-25T09:30:08.686Z	INFO	log/reporter.go:40	2021-10-25T09:30:08Z - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-10-25T09:30:09.371Z	INFO	cmd/enroll_cmd.go:701	Fleet Server - Starting
2021-10-25T09:30:15.382Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Starting
2021-10-25T09:30:21.394Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Starting
2021-10-25T09:30:27.405Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Starting
2021-10-25T09:30:33.416Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Starting
2021-10-25T09:30:39.430Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Starting
2021-10-25T09:30:45.441Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Starting
2021-10-25T09:30:51.455Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Starting
2021-10-25T09:30:57.465Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Starting
2021-10-25T09:31:03.479Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Starting
2021-10-25T09:31:09.491Z	INFO	cmd/enroll_cmd.go:706	Fleet Server - Starting
2021-10-25T09:31:11.834Z	WARN	status/reporter.go:236	Elastic Agent status changed to: 'degraded'
2021-10-25T09:31:11.834Z	INFO	log/reporter.go:40	2021-10-25T09:31:11Z - message: Application: fleet-server--7.14.0[]: State changed to DEGRADED: Missed last check-in - type: 'STATE' - sub_type: 'RUNNING'
2021-10-25T09:31:12.497Z	INFO	cmd/enroll_cmd.go:682	Fleet Server - Missed last check-in
2021-10-25T09:31:12.497Z	WARN	[tls]	tlscommon/tls_config.go:98	SSL/TLS verifications disabled.
2021-10-25T09:31:12.555Z	INFO	cmd/enroll_cmd.go:414	Starting enrollment to URL: http://localhost:8220/
2021-10-25T09:31:12.659Z	INFO	cmd/run.go:189	Shutting down Elastic Agent and sending last events...

system · November 22, 2021, 11:33am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.