Dear Elastic Community,
We're installing a standalone elastic agent as a fleet server on a server dedicated for it in an air-gapped environment. We use APIs instead of Kibana UI for creating a policy and a service token, as the installation process needs to be automated with ansible.
Our customer does not allow us the installation from TAR-files. Therefore, elastic agent is available to us only as an RPM package.
We've installed the agent in version 8.11.3 with dnf (the same version as the ELK-Stack we use). We registered the service and configured it with elastic-agent.yml:
outputs:
default:
type: elasticsearch
hosts: ["https://elasticsearch_retracted_FQDN:elasticsearch_retracted_port"]
protocol: https
username: "retracted_username"
password: "retracted_password"
ssl:
enabled: true
certificate_authorities: "/retracted/path/to/the/certificate/of/the/certificate/authority/of/elasticsearch"
verification_mode: full
inputs:
- type: fleet-server
policy.id: 'fleet-server-fleet_server-5d8347d3-9f9a-4e22-9ec8-xxxxxxxxxxxx'
server:
ssl:
enabled: true
certificate: "/retracted/path/to/the/certificate"
key: "/retracted/path/to/the/certificate"
key_passphrase: "/path/to/the/file/with/the/key/passphrase.txt"
fleet:
kibana:
hosts: ["https://kibana_retracted_FQDN_1:kibana_retracted_port", "https://kibana_retracted_FQDN_2:kibana_retracted_port"]
ssl.certificate_authorities: ["/retracted/path/to/the/certificate/of/the/certificate/authority/of/kibana"]
protocol: "https"
service_token: "retracted_token"
ssl.verification_mode: full
agent.logging.to_stderr: true
agent.logging.to_files: true
agent.logging.files:
path: /var/log/elastic-agent
The configuration of the service:
[Unit]
Description=Agent manages other beats based on configuration provided.
Documentation=https://www.elastic.co/beats/elastic-agent
Wants=network-online.target
After=network-online.target[Service]
UMask=0027
Environment="GODEBUG='madvdontneed=1'"
Environment="BEAT_CONFIG_OPTS=-c /etc/elastic-agent/elastic-agent.yml"
ExecStart=/usr/bin/elastic-agent run --environment systemd $BEAT_CONFIG_OPTS
Restart=always# By setting KillMode=process, we ensure that in a scenario where the upgraded Agent's
# process is repeatedly crashing, systemd keeps the Upgrade Watcher process running so
# it can monitor the Agent process for long enough to initiate a rollback.
# See also Prevent `systemd` from killing the Upgrade Watcher process after the main Agent process has crashed by ycombinator · Pull Request #3220 · elastic/elastic-agent · GitHub.
KillMode=process[Install]
WantedBy=multi-user.target
We started the service per systemctl. However, it reports the error:
Mar 14 15:27:01 ile76236 elastic-agent[2240407]: {"log.level":"error","@timestamp":"2024-03-14T15:27:01.315+0100","log.origin":{"file.name":"coordinator/coordinator.go","file.line":557},"message":"Unit state changed fleet-server-default (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: elastic fail 401: security_exception: missing authentication credentials for REST request [/]","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
Mar 14 15:27:01 ile76236 elastic-agent[2240407]: {"log.level":"error","@timestamp":"2024-03-14T15:27:01.315+0100","log.origin":{"file.name":"coordinator/coordinator.go","file.line":557},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: elastic fail 401: security_exception: missing authentication credentials for REST request [/]","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
Mar 14 15:27:01 ile76236 elastic-agent[2240407]: {"log.level":"error","@timestamp":"2024-03-14T14:27:01.305Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"elastic fail 401: security_exception: missing authentication credentials for REST request [/]","ecs.version":"1.6.0"}
Mar 14 15:27:01 ile76236 elastic-agent[2240407]: {"log.level":"info","@timestamp":"2024-03-14T14:27:01.305Z","message":"Stats endpoint (/opt/elastic/agent/elastic-agent/data/tmp/UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock) finished: accept unix /opt/elastic/agent/elastic-agent/data/tmp/UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock: use of closed network connection","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","log.logger":"fleet-metrics.api","ecs.version":"1.6.0"}
Mar 14 15:27:01 ile76236 elastic-agent[2240407]: {"log.level":"info","@timestamp":"2024-03-14T14:27:01.305Z","message":"Error - failed version compatibility check with elasticsearch: elastic fail 401: security_exception: missing authentication credentials for REST request [/]","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"FAILED","ecs.version":"1.6.0"}
Mar 14 15:27:01 ile76236 elastic-agent[2240407]: {"log.level":"error","@timestamp":"2024-03-14T14:27:01.305Z","message":"Fleet Server failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"failed version compatibility check with elasticsearch: elastic fail 401: security_exception: missing authentication credentials for REST request [/]","ecs.version":"1.6.0"}
Could you please help us in resolving the issue?