Hi Folks,
we are running a ELK Stack in version 7.16 with several elastic-agents managed by the fleet server.
After we upgraded to 7.16 the elastic-agents needs to be upgraded also.
Some upgrades for several elastic-agents over the fleet management are running properly.
But some upgrades dont work.
After triggering the upgrade via webGUI, the elastic-agents tries to download and upgrade the installation, but stopped with errors.
I have the following logs:
{"log.level":"info","@timestamp":"2022-01-18T16:18:19.573Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-01-18T17:18:19+01:00 - message: Application: [bb08e522-37df-49fc-8ec9-d459b2c9d36e]: State changed to UPDATING: Update to version '7.16.3' started - type: 'STATE' - sub_type: 'UPDATING'","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-01-18T16:20:25.392Z","log.origin":{"file.name":"remote/client.go","file.line":169},"message":"Request method: POST, path: /api/fleet/agents/bb08e522-37df-49fc-8ec9-d459b2c9d36e/acks","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-01-18T16:20:26.419Z","log.origin":{"file.name":"fleet/fleet_acker.go","file.line":66},"message":"action with id '2c260970-aee1-40db-b374-2f391a62867c' was just acknowledged","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-01-18T16:20:26.419Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-01-18T17:20:26+01:00 - message: Application: [bb08e522-37df-49fc-8ec9-d459b2c9d36e]: State changed to FAILED: failed verification of agent binary: 2 errors occurred:\n\t* fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc: no such file or directory\n\t* fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc: failed loading public key: Get \"https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n\n - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-01-18T16:20:26.419Z","log.origin":{"file.name":"dispatcher/dispatcher.go","file.line":95},"message":"Failed to dispatch action 'action_id: 2c260970-aee1-40db-b374-2f391a62867c, type: UPGRADE', error: failed verification of agent binary: 2 errors occurred:\n\t* fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc: no such file or directory\n\t* fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc: failed loading public key: Get \"https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n\n","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-01-18T16:20:26.419Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":180},"message":"failed to dispatch actions, error: failed verification of agent binary: 2 errors occurred:\n\t* fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc: no such file or directory\n\t* fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc: failed loading public key: Get \"https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n\n","ecs.version":"1.6.0"}
When i try to download the file manualy, it works.
wget https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc
--2022-01-18 17:32:26-- https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.16.3-linux-x86_64.tar.gz.asc
Resolving artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
Connecting to artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 488 [binary/octet-stream]
Saving to: ‘elastic-agent-7.16.3-linux-x86_64.tar.gz.asc’
elastic-agent-7.16.3-linux-x86_64.tar.gz.asc 100%[====================================================================================================================================>] 488 --.-KB/s in 0s
2022-01-18 17:32:26 (283 MB/s) - ‘elastic-agent-7.16.3-linux-x86_64.tar.gz.asc’ saved [488/488]
Do you have any ideas?
A manual upgrade via deinstall and install again works fine. But we had this problem with every version.
Best regards
winter