Unable to use mounted elastic-agent.yml if running in elastic-agent container

[hendry.lim (Hendry)]

Hi,

I am using elastic-agent-7.9.0-SNAPSHOT docker image (also tested and encountered the same error with elastic-agent-8.0.0-SNAPSHOT docker image).
My docker-compose.yml:

version: '3.7'
services:
  elastic-agent:
    image: docker.elastic.co/beats/elastic-agent:7.9.0-SNAPSHOT
    container_name: elastic-agent
    command: ["enroll", "http://kibana:5601", "<enrollment_token>"]
    environment:
      - "FLEET_ENROLL=1"
      - "FLEET_SETUP=1"
      - "KIBANA_HOST=http://kibana:5601"
    volumes:
      - ./elastic-agent.yml:/usr/share/elastic-agent/elastic-agent.yml

Error message:

fail to enroll: could not save enrollment information: could not backup /usr/share/elastic-agent/elastic-agent.yml: rename /usr/share/elastic-agent/elastic-agent.yml /usr/share/elastic-agent/elastic-agent.yml.2020-06-29T23-51-39.7743.bak: device or resource busy

This error leads to another issue: I am not able to specify the ca.crt for Metricbeat and Filebeat running in the elastic-agent container.

This also breaks if ES requires authentication, because I am not able to specify ES username and password in elastic-agent.yml for Metricbeat and Filebeat.

[ruflin]

Hi @hendry.lim Thanks for trying out the Elastic Agent. Based on your error, I assume you are using the Elastic Agent together with the Ingest Manager and tried to enroll the agent?

The error indicates, that the elastic agent is missing the permissions to rename / backup the config. He needs to have these permissions because when enrolling into Ingest Manager, the existing config is backed up and overwritten by the config coming from Ingest Manager.

If you are using Ingest Manager, you don't need to copy your own configuration into the container. Also username / password are replaced by API Keys when used with Ingest Manager, so no need to specify it.

Did you perhaps wanted to run Agent in standalone mode? If yes, you need to remove the enrollment part.

[hendry.lim (Hendry)]

Hi @ruflin Thank you for your prompt reply.

Based on your error, I assume you are using the Elastic Agent together with the Ingest Manager and tried to enroll the agent?

That is correct. I am trying out Elastic Agent with Ingest Manager and Fleet. I am able to get the Elastic Agent to run in container, but Filebeat and Metricbeat are unable to connect back to ES due to SSL error caused by using self-signed certificate.
How do I configure ca.crt or ca_sha256 for Filebeat and Metricbeat running in the same Elastic Agent container?

Ping request failed with: Get https://es01:9200: x509: certificate signed by unknown authority

I just found out that I am unable to edit the Elastic Agent configuration data source with the latest 7.9.0 snapshot. It's the same for a new configuration.

[hendry.lim (Hendry)]

Saw this issue #19504 raised, hopefully this will be the solution for the custom certificate authority use case.

[ruflin]

Yes, that should solve it. @pierhugues Can you think of a temporary workaround?

[sej7278]

elastic-agent-7.8.0-x86_64.rpm and elastic-agent-7.8.0-linux-x86_64.tar.gz have the same issue, setting ssl.certificate_authorities in elastic-agent.yml didn't work as it does for the beats when they're not under the agent.

[hendry.lim (Hendry)]

The latest elastic-agent-7.9.0-SNAPSHOT seems to be broken.

# docker-compose.yml
version: '3.7'
services:
  elastic-agent:
    image: docker.elastic.co/beats/elastic-agent:7.9.0-SNAPSHOT
    container_name: elastic-agent
    environment:
      - "FLEET_ENROLL=1"
      - "KIBANA_HOST=http://kibana:5601"
      - "FLEET_ENROLLMENT_TOKEN=<token>"

Errors:

Error: accepts 2 arg(s), received 1
Usage:
  elastic-agent enroll <kibana_url> <enrollment_token> [flags]

Flags:
  -p, --ca_sha256 string                 Comma separated list of certificate authorities hash pins used for certificate verifications
  -a, --certificate_authorities string   Comma separated list of root certificate for server verifications
  -f, --force                            Force overwrite the current and do not prompt for confirmation
  -h, --help                             help for enroll

Global Flags:
  -c, -- string              Configuration file, relative to path.config (default "elastic-agent.yml") (default "elastic-agent.yml")
      --path.config string   Configuration path (default "${path.home}")
      --path.data string     Data path contains Agent managed binaries (default "/usr/share/elastic-agent/data")
      --path.home string     Agent root path (default "/usr/share/elastic-agent")
      --strict.perms         Strict permission checking on config files (default true)

accepts 2 arg(s), received 1

It still works if I use command: ["enroll", "http://kibana:5601", "<token>"]
instead of using FLEET_ENROLLMENT_TOKEN.