Beats in elastic-agent reporting "failed to connect to backoff"

Elastic Stack Beats
,
3

I found a work around, but still am working to find out why this was necessary.

As noted earlier in the Elastic/Agent/data/elastic-agent-*/logs/default/filebeat-json.log file (and the metricbeat-json.log file) I was seeing this message repeating.

{"log.level":"error","@timestamp":"2022-03-15T17:03:41.086Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":154},"message":"Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get \"http://localhost:9200\": dial tcp [::1]:9200: connect: connection refused","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-15T17:03:41.086Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":145},"message":"Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 94 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-15T17:03:41.086Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":219},"message":"retryer: send unwait signal to consumer","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-15T17:03:41.086Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":223},"message":"  done","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-03-15T17:03:41.086Z","log.logger":"esclientleg","log.origin":{"file.name":"transport/logging.go","file.line":37},"message":"Error dialing dial tcp [::1]:9200: connect: connection refused","service.name":"filebeat","network":"tcp","address":"localhost:9200","ecs.version":"1.6.0"}

In my Fleet Settings I had specified that the Elasticsearch hosts URL was https://:9200 but in these message it shows that it is trying to connect to http://localhost:9200

I confirmed that the fleet.yml file was correct. The only thing in the elastic-agent.yml file was

fleet:
  enabled: true

Therefore I would expect that the filebeat and metricbeat ymls would be picking up the fleet configuration.

Looking at the the filebeat.yml file (at Elastic/Agent/data/elastic-agent-*/install/filebeat-7.17.0-linux-x86_64 I found this:

 ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

I updated that file to this:

 ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["<nlb dns name:9200"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "<actual password"
  ssl.verification_mode: none

After that update and restarting the elastic-agent, data began appearing in the data streams for the OS & System integrations that I had added to the policy for that host.

So, the question is, why isn't filebeat picking up the configuration from the fleet settings?