Fleet Server installation options

Elastic Stack Kibana
1

Hey,

currently, the only way to install fleet server is using a subprocess of elastic agent. This is very weird to me - Elastic Agent is a client component, and Fleet Server is an infrastructure component.

I typically run applications in Docker, so that they're isolated and portable. Afaik, there is currently also no Elastic Agent Docker Image.

Are there any plans to support a more traditional deployment model for the Fleet Server, such as a Docker container, like there is for APM server?

2

H @nemhods There is a docker image for Elastic Agent, you can find it linked here: Run Elastic Agent on Kubernetes | Fleet User Guide [7.13] | Elastic The image also contains the fleet-server input.

Some details on how to run it you can find here: Fleet server shutsdown after successfully enrolling agent · Issue #401 · elastic/fleet-server · GitHub The problem at the moment is lack of documentation on our end but we plan to follow up on this quickly.

For inspiration, here is a docker compose file we use internally for some test environemnts: elastic-package/static_snapshot_yml.go at 1a5b4118de7125b9403c973120f0e37a4694320c · elastic/elastic-package · GitHub

1 Like
3

Looks good, thank you!
I'm not sure how I missed the Elastic-Agent Docker image...

I notice that for your tests, you don't give the Fleet Server any persistence options (like Docker Volumes). Is it okay to do it that way in practice as well? It would mean that the Fleet Server re-enrolls every time the machine is rebooted.

4

For further reference, while others may also be waiting for documentation:

This is a working docker-compose for an almost proper setup with HTTPS.

services:
  fleet_server:
    image: docker.elastic.co/beats/elastic-agent:7.13.0
    hostname: "fleet_hostname"
    environment:
      - FLEET_SERVER_ENABLE=1
      - FLEET_SERVER_INSECURE_HTTP=0
      - FLEET_SERVER_SERVICE_TOKEN=REPLACE_ME
      - FLEET_SERVER_CERT=/usr/share/elastic-agent/cert.crt
      - FLEET_SERVER_CERT_KEY=/usr/share/elastic-agent/cert.key
      - FLEET_URL=REPLACE_ME
      - FLEET_SERVER_ELASTICSEARCH_HOST=REPLACE_ME
      - FLEET_INSECURE=1
      - FLEET_SERVER_ELASTICSEARCH_CA=/usr/share/elastic-agent/ca.pem
      - KIBANA_FLEET_CA=/usr/share/elastic-agent/ca.pem
      - FLEET_CA=/usr/share/elastic-agent/ca.pem
      - ELASTICSEARCH_CA=/usr/share/elastic-agent/ca.pem
      - KIBANA_CA=/usr/share/elastic-agent/ca.pem
      - FLEET_CA=/usr/share/elastic-agent/ca.pem
    ports:
      - "8220:8220"
    healthcheck:
      test: "curl -f http://127.0.0.1:8220/api/status | grep HEALTHY 2>&1 >/dev/null"
      retries: 12
      interval: 5s
    volumes:
      - ./cert.crt:/usr/share/elastic-agent/cert.crt:ro
      - ./cert.key:/usr/share/elastic-agent/cert.key:ro
      - ./ca.pem:/usr/share/elastic-agent/ca.pem:ro

I had to set FLEET_INSECURE=1 because no matter what I tried, the fleet server would always fail with

fleet_server_1  | 2021-05-28T19:39:13.417Z      INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
fleet_server_1  | 2021-05-28T19:39:13.417Z      INFO    cmd/run.go:197  Shutting down completed.
fleet_server_1  | 2021-05-28T19:39:13.417Z      INFO    log/reporter.go:40      2021-05-28T19:39:13Z - message: Application: fleet-server--7.13.0[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
fleet_server_1  | 2021-05-28T19:39:13.418Z      INFO    [api]   api/server.go:66        Stats endpoint (/usr/share/elastic-agent/state/data/tmp/elastic-agent.sock) finished: accept unix /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock: use of closed network connection
fleet_server_1  | Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
fleet_server_1  | Error: enrollment failed: exit status 1

In my environment, all certificates are directly signed by a single CA, and I provided that CA to every CA-Option available to elastic-agent. The certificates for the fleet server are naturally also from that CA. It did manage to contact Elasticsearch, but then seems to fail in a step where it tries to connect to itself? At least that's what I get from "fail to execute request to fleet-server". I don't know, and I also couldn't find a quick way to enable debug logging for a dockerized elastic agent.

All possible env vars are (from ./elastic-agent container -h):

The following actions are possible and grouped based on the actions.

* Elastic Agent Fleet Enrollment
  This enrolls the Elastic Agent into a Fleet Server. It is also possible to have this create a new enrollment token
  for this specific Elastic Agent.

  FLEET_ENROLL - set to 1 for enrollment into fleet-server. If not set, Elastic Agent is run in standalone mode.
  FLEET_URL - URL of the Fleet Server to enroll into
  FLEET_ENROLLMENT_TOKEN - token to use for enrollment. This is not needed in case FLEET_SERVER_ENABLED and FLEET_ENROLL is set. Then the token is fetched from Kibana.
  FLEET_CA - path to certificate authority to use with communicate with Fleet Server [$KIBANA_CA]
  FLEET_INSECURE - communicate with Fleet with either insecure HTTP or unverified HTTPS

  The following vars are need in the scenario that Elastic Agent should automatically fetch its own token.

  KIBANA_FLEET_HOST - kibana host to enable create enrollment token on [$KIBANA_HOST]
  KIBANA_FLEET_USERNAME - kibana username to create enrollment token [$KIBANA_USERNAME]
  KIBANA_FLEET_PASSWORD - kibana password to create enrollment token [$KIBANA_PASSWORD]
  FLEET_TOKEN_NAME - token name to use for fetching token from Kibana. This requires Kibana configs to be set.
  FLEET_TOKEN_POLICY_NAME - token policy name to use for fetching token from Kibana. This requires Kibana configs to be set.

* Bootstrapping Fleet Server
  This bootstraps the Fleet Server to be run by this Elastic Agent. At least one Fleet Server is required in a Fleet
  deployment for other Elastic Agent to bootstrap. In case the Elastic Agent is run without fleet-server. These variables
  are not needed.

  If FLEET_SERVER_ENABLE and FLEET_ENROLL is set but no FLEET_ENROLLMENT_TOKEN, the token is automatically fetched from Kibana.

  FLEET_SERVER_ENABLE - set to 1 enables bootstrapping of Fleet Server inside Elastic Agent (forces FLEET_ENROLL enabled)
  FLEET_SERVER_ELASTICSEARCH_HOST - elasticsearch host for Fleet Server to communicate with [$ELASTICSEARCH_HOST]
  FLEET_SERVER_ELASTICSEARCH_USERNAME - elasticsearch username for Fleet Server [$ELASTICSEARCH_USERNAME]
  FLEET_SERVER_ELASTICSEARCH_PASSWORD - elasticsearch password for Fleet Server [$ELASTICSEARCH_PASSWORD]
  FLEET_SERVER_ELASTICSEARCH_CA - path to certificate authority to use with communicate with elasticsearch [$ELASTICSEARCH_CA]
  FLEET_SERVER_SERVICE_TOKEN - service token to use for communication with elasticsearch
  FLEET_SERVER_POLICY_ID - policy ID for Fleet Server to use for itself ("Default Fleet Server policy" used when undefined)
  FLEET_SERVER_HOST - binding host for Fleet Server HTTP (overrides the policy). By default this is 0.0.0.0.
  FLEET_SERVER_PORT - binding port for Fleet Server HTTP (overrides the policy)
  FLEET_SERVER_CERT - path to certificate to use for HTTPS endpoint
  FLEET_SERVER_CERT_KEY - path to private key for certificate to use for HTTPS endpoint
  FLEET_SERVER_INSECURE_HTTP - expose Fleet Server over HTTP (not recommended; insecure)

* Preparing Kibana for Fleet
  This prepares the Fleet plugin that exists inside of Kibana. This must either be enabled here or done externally
  before Fleet Server will actually successfully start. All the Kibana variables are not needed in case Elastic Agent
  should not setup Fleet. To manually trigger KIBANA_FLEET_SETUP navigate to Kibana -> Fleet -> Agents and enabled it.

  KIBANA_FLEET_SETUP - set to 1 enables the setup of Fleet in Kibana by Elastic Agent. This was previously FLEET_SETUP.
  KIBANA_FLEET_HOST - Kibana host accessible from fleet-server. [$KIBANA_HOST]
  KIBANA_FLEET_USERNAME - kibana username to enable Fleet [$KIBANA_USERNAME]
  KIBANA_FLEET_PASSWORD - kibana password to enable Fleet [$KIBANA_PASSWORD]
  KIBANA_FLEET_CA - path to certificate authority to use with communicate with Kibana [$KIBANA_CA]
  KIBANA_REQUEST_RETRY_SLEEP - specifies sleep duration taken when agent performs a request to kibana [default 1s]
  KIBANA_REQUEST_RETRY_COUNT - specifies number of retries agent performs when executing a request to kibana [default 30]

The following environment variables are provided as a convenience to prevent a large number of environment variable to
be used when the same credentials will be used across all the possible actions above.

  ELASTICSEARCH_HOST - elasticsearch host [http://elasticsearch:9200]
  ELASTICSEARCH_USERNAME - elasticsearch username [elastic]
  ELASTICSEARCH_PASSWORD - elasticsearch password [changeme]
  ELASTICSEARCH_CA - path to certificate authority to use with communicate with elasticsearch
  KIBANA_HOST - kibana host [http://kibana:5601]
  KIBANA_USERNAME - kibana username [$ELASTICSEARCH_USERNAME]
  KIBANA_PASSWORD - kibana password [$ELASTICSEARCH_PASSWORD]
  KIBANA_CA - path to certificate authority to use with communicate with Kibana
5

Thanks for sharing all the details also for others here! @blaker Could you take a look at the CA issues?

For the persistence: Both ways should work. There is one issue we are currently working on for the fresh enrollment each time: The list of Elastic Agent in Fleet keeps getting longer and is not automatically cleaned up. The concept we have in mind to solve this is ephemeral agents which is agent which automatically are unenrolled after some time of inactivity.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.