Elastic agent failing to enroll on a fresh install


I am having trouble getting an elastic agent to enroll on a host for monitoring (more specific details to come below). As an overview, here's what I'm working with:

  • Freshly installed Elasticsearch and kibana (both 8.0.1) installed on the same Ubuntu server host

  • Freshly installed elastic agent (8.0.1) on it's own Ubuntu server host functioning as a fleet server. It is checked-in and healthy in Kibana. Both of these hosts have freshly installed operating systems as well.

  • A third debian host running Apache in which I have downloaded elastic agent for monitoring. This is the one that I am attempting to enroll, but am not having any luck.

My kibana.yml looks like this:

### >>>>>>> BACKUP START: Kibana interactive setup (2022-03-05T16:41:22.308Z)

# For more configuration options see the configuration guide for Kibana in
# https://www.elastic.co/guide/index.html

# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: ""

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# Defaults to `false`.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576

# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"

# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://localhost:9200"]

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Kibana can also authenticate to Elasticsearch via "service account tokens".
# Service account tokens are Bearer style tokens that replace the traditional username/password based configuration.
# Use this token instead of a username/password.
# elasticsearch.serviceAccountToken: "my_token"

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# =================== System: Elasticsearch (Optional) ===================
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# =================== System: Logging ===================
# Set the value of this setting to off to suppress all logging output, or to debug to log everything. Defaults to 'error'
#logging.root.level: debug

# Enables you to specify a file where Kibana stores log output.
#  appenders:
#    file:
#      type: file
#      fileName: /var/log/kibana/kibana.log
#      layout:
#        type: json
#  root:
#    appenders:
#      - default
#      - file
#  layout:
#    type: json

# Logs queries sent to Elasticsearch.
#  - name: elasticsearch.query
#    level: debug

# Logs http responses.
#  - name: http.server.response
#    level: debug

# Logs system usage information.
#  - name: metrics.ops
#    level: debug

# =================== System: Other ===================
# The path where Kibana stores persistent data not saved in Elasticsearch. Defaults to data
#path.data: data

# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

# =================== Frequently used (Optional)===================

# =================== Saved Objects: Migrations ===================
# Saved object migrations run at startup. If you run into migration-related issues, you might need to adjust these settings.

# The number of documents migrated at a time.
# If Kibana can't start up or upgrade due to an Elasticsearch `circuit_breaking_exception`,
# use a smaller batchSize value to reduce the memory pressure. Defaults to 1000
# migrations.batchSize: 1000

# The maximum payload size for indexing batches of upgraded saved objects.
# To avoid migrations failing due to a 413 Request Entity Too Large response from Elasticsearch.
# This value should be lower than or equal to your Elasticsearch cluster’s `http.max_content_length`
# configuration option. Default: 100mb
# migrations.maxBatchSizeBytes: 100mb

# The number of times to retry temporary migration failures. Increase the setting
# if migrations fail frequently with a message such as `Unable to complete the [...] step after
# 15 attempts, terminating`. Defaults to 15
# migrations.retryAttempts: 15

# =================== Search Autocomplete ===================
# Time in milliseconds to wait for autocomplete suggestions from Elasticsearch.
# This value must be a whole number greater than zero. Defaults to 1000
# data.autocomplete.valueSuggestions.timeout: 1000

# Maximum number of documents loaded by each shard to generate autocomplete suggestions.
# This value must be a whole number greater than zero. Defaults to 100000
# data.autocomplete.valueSuggestions.terminateAfter: 100000

### >>>>>>> BACKUP END: Kibana interactive setup (2022-03-05T16:41:22.308Z)

# This section was automatically generated during setup.
server.port: 5601
elasticsearch.hosts: ['']
logging.appenders.file.type: file
logging.appenders.file.fileName: /var/log/kibana/kibana.log
logging.appenders.file.layout.type: json
logging.root.appenders: [default, file]
pid.file: /run/kibana/kibana.pid
elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE2NDY0OTg0ODE5NjM6YTRoRGpMdWNUVTJweWdqRnlLYTJDUQ
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1646498482303.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: [''], ca_trusted_fingerprint: fbe6ff36da03a7ab237b4ff4ae2125a2d7a9f270e5765e44f3eb2ac9803b2ec4}]

Elasticsearch.yml looks like this:

# ======================== Elasticsearch Configuration =========================
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
# ---------------------------------- Cluster -----------------------------------
# Use a descriptive name for your cluster:
#cluster.name: my-application
# ------------------------------------ Node ------------------------------------
# Use a descriptive name for the node:
#node.name: node-1
# Add custom attributes to the node:
#node.attr.rack: r1
# ----------------------------------- Paths ------------------------------------
# Path to directory where to store the data (separate multiple locations by comma):
path.data: /var/lib/elasticsearch
# Path to log files:
path.logs: /var/log/elasticsearch
# ----------------------------------- Memory -----------------------------------
# Lock the memory on startup:
#bootstrap.memory_lock: true
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
# Elasticsearch performs poorly when the system is swapping the memory.
# ---------------------------------- Network -----------------------------------
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#http.port: 9200
# For more information, consult the network module documentation.
# --------------------------------- Discovery ----------------------------------
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["", "[::1]"]
#discovery.seed_hosts: ["host1", "host2"]
# Bootstrap the cluster using an initial set of master-eligible nodes:
#cluster.initial_master_nodes: ["node-1", "node-2"]
# For more information, consult the discovery and cluster formation module documentation.
# ---------------------------------- Various -----------------------------------
# Allow wildcard deletion of indices:
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 05-03-2022 16:19:00
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["elasticserver"]

# Allow HTTP API connections from localhost and local networks
# Connections are encrypted and require user authentication
http.host: [_local_, _site_]

# Allow other nodes to join the cluster from localhost and local networks
# Connections are encrypted and mutually authenticated
#transport.host: [_local_, _site_]

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

Finally, when I try to run sudo ./elastic-agent install --url= --enrollment-token=ZE8zMVduOEJ1Wjc2NFJDSWdLM0U6a0NNUWNRZkhUSG1WbHpSSWxlQzRvQQ==

The below is printed:

Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2022-03-05T18:01:28.022Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":456},"message":"Starting enrollment to URL:","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: http: server gave HTTP response to HTTPS client
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.0/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.0/fleet-troubleshooting.html

I have reviewed the troubleshooting guide and attempted manual enroll with ./elastic-agent install -f but I'm not having luck down that path either. There didn't appear to much else relevant in that document.

It would seem to me that there is some kind of certificate related issue here, but I'm not sure what further troubleshooting steps would be appropriate at this point. I'm hoping there's just some easy oversight I made some where, but I did attempt this on a separate windows host with similar results. Any hints or advice would be greatly appreciated it!


I would like to add that, by instead running this:

 ./elastic-agent install --url= --enrollment-token=ZE8zMVduOEJ1Wjc2NFJDSWdLM0U6a0NNUWNRZkhUSG1WbHpSSWxlQzRvQQ== --insecure

(the differences is the dropped s in the protocol, so just http, and the addition of "--insecure")

I get

Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.

Unfortunately, log data doesn't appear to be coming in, and this agent is indicated as unhealthy

Found the fix for this after starting over from scratch and meticulously paying attention to detail. For the life of me I couldn't understand this output Error: fail to enroll: fail to execute request to fleet-server: http: server gave HTTP response to HTTPS client when security is auto-configured on the install. I was doubly confused as to why, when adding a new agent, they would at least appear to check-in with unhealthy status and not actually send any logs.

Well, the problem wasn't with Elasticsearch, or kibana. Rather, it was a fleet server issue (which makes sense in hindsight).

Here's what I did:

When enrolling a fleet server using the Quick start option, the Start Fleet Server command presented in the UI is:

sudo ./elastic-agent install  \
  --fleet-server-es=https://<IP>:9200 \
  --fleet-server-service-token=<token> \
  --fleet-server-policy=499b5aa7-d214-5b5d-838b-3cd76469844e \
  --fleet-server-es-ca-trusted-fingerprint=<fingerprint> \

In order to resolve these problems, one must remove --fleet-server-insecure-http from this command. Then, for each agent to be added, add --insecure to the enrollment command as noted in the troubleshooting guide. Then, agents will check-in as healthy and actually send data.

It would be helpful if Elastic could remove --fleet-server-insecure-http from the command offered when using the quick start option to help out with this. Unless someone really needs it to be in the clear, which I would imagine to be edge cases only, I at least don't see a common enought point to it because following the method I detailed above will still allow the traffic to be encrypted, just the cert isn't verified.

1 Like

Thank you very much for the post and answer @epheria! I've been struggling with the same exact issue and your recommendation solved the issue.

To help future Googlers, the key error message that I was getting: http: server gave HTTP response to HTTPS client. In short, the elastic-agent acting as the fleet server was replying back in HTTP to new the new elastic-agents I was trying to enroll as HTTPs.

Behind the scenes, it looks like the --fleet-server-insecure-http modifies the fleet.yml file of the elastic-agent acting as the fleet server. Below is a redact snippet of a working HTTPS fleet server for reference. For instance, notice the in the fleet section that the protocol: https, in addition to the need to specify certificates for the actual encryption.

   id: <VALID_ID>
    enabled: false
    host: ""
    port: 6791
  enabled: true
  access_api_key: <VALID_API_KEY>
  protocol: https
  host: <VALID_PUBLIC_HOSTNAME>:8220		# But not localhost!
    verification_mode: ""
    - |
      -----BEGIN CERTIFICATE-----
      -----END CERTIFICATE-----
    renegotiation: never
  timeout: 10m0s
  proxy_disable: true
    threshold: 10000
    check_frequency_sec: 30
    id: ""
      id: fleet-server-policy
        protocol: https
        service_token: <VALID_SERVICE_TOKEN>
          verification_mode: ""
          renegotiation: never
          ca_trusted_fingerprint: <VALID_FINGERPRINT>
        proxy_disable: false
        proxy_headers: {}
    port: 8220
    internal_port: 8221
      verification_mode: ""
      certificate: |
        -----BEGIN CERTIFICATE-----
        -----END CERTIFICATE-----
      key: |
        -----BEGIN RSA PRIVATE KEY-----
        -----END RSA PRIVATE KEY-----
      renegotiation: never

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.