Upgrade Agents Offline

[scottdfedorov (Scott Fedorov)]

The docs are clear that Fleet:

Requires internet access for Kibana to download integration packages

and

No proxy support in Elastic Agent

However, it appears that each Agent also requires internet access in order to upgrade.
I just upgraded to 7.12.1 and am attempting to upgrade my Agents, but am getting the following error:

[elastic_agent][error] failed to dispatch actions, error: failed upgrade of agent binary: 2 errors occurred:
* package 'C:\Program Files\Elastic\Agent\data\elastic-agent-9b2fec\downloads\elastic-agent-7.12.1-windows-x86_64.zip.sha512' not found: open C:\Program Files\Elastic\Agent\data\elastic-agent-9b2fec\downloads\elastic-agent-7.12.1-windows-x86_64.zip.sha512: The system cannot find the file specified.
* fetching package failed: Get "https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.12.1-windows-x86_64.zip": dial tcp 34.120.127.130:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Is this correct?
I know there are some open GH issues to allow for offline Fleet, but seems to apply to the Kibana connecting to epr. Not sure if also applies to upgrades to Agents. I was hoping the Agent would get the upgrade packages from Kibana and not download themselves.

Also, if this is the case, hopefully the docs on upgrading agents can be amended to include the requirement that the Agent be able to access the internet.

Is there a workaround? I'm thinking perhaps we can copy the artifact to the folder that the agent is looking in, if it will just pick it up. Something to try next....

I did try setting the internet proxy on the server, of course to no avail. (Expected)

[scottdfedorov (Scott Fedorov)]

Meant to post the previous on Friday, but apparently it didn't want to post for some reason.

I tried testing again this morning, and got the same error.
I did notice that in the Program Files\Elastic\Agent\data\elastic-agent-9b2fec\downloads folder there was already a version of 7.12 in there, but as the error noted, it was missing the .sha512 file.

I tried adding the sha and asc files from the download page for the Agent, and ran it again, and it failed to upgrade with a new error.

 hash for 'elastic-agent-7.12.1-windows-x86_64.zip' not found in 'C:\Program Files\Elastic\Agent\data\elastic-agent-9b2fec\downloads\elastic-agent-7.12.1-windows-x86_64.zip.sha512'
	* hash for 'elastic-agent-7.12.1-windows-x86_64.zip' not found

Log entries below.

Now the previous download of the 7.12 file is missing in the folder so the process removed the downloaded files. I then tried running again and back to getting the original error, and the downloaded zip is present again.

So I'm wondering if there's an issue with getting the sha file? Strange..

08:49:22.532
info
elastic_agent
C:\Program Files\Elastic\Agent\data\elastic-agent-9b2fec\logs\elastic-agent-json.log
[elastic_agent][info] 2021-05-10T13:49:22Z: type: 'STATE': sub_type: 'UPDATING' message: Application: [f8192350-aeb6-11eb-bcaf-6542f096436e]: State changed to UPDATING: Update to version '7.12.1' started
08:49:24.804
error
elastic_agent
C:\Program Files\Elastic\Agent\data\elastic-agent-9b2fec\logs\elastic-agent-json.log
[elastic_agent][error] 2021-05-10T13:49:24Z: type: 'ERROR': sub_type: 'FAILED' message: Application: [f8192350-aeb6-11eb-bcaf-6542f096436e]: State changed to FAILED: failed verification of agent binary: 2 errors occurred:
	* hash for 'elastic-agent-7.12.1-windows-x86_64.zip' not found in 'C:\Program Files\Elastic\Agent\data\elastic-agent-9b2fec\downloads\elastic-agent-7.12.1-windows-x86_64.zip.sha512'
	* hash for 'elastic-agent-7.12.1-windows-x86_64.zip' not found
08:49:24.804
error
elastic_agent
C:\Program Files\Elastic\Agent\data\elastic-agent-9b2fec\logs\elastic-agent-json.log
[elastic_agent][error] failed to dispatch actions, error: failed verification of agent binary: 2 errors occurred:
	* hash for 'elastic-agent-7.12.1-windows-x86_64.zip' not found in 'C:\Program Files\Elastic\Agent\data\elastic-agent-9b2fec\downloads\elastic-agent-7.12.1-windows-x86_64.zip.sha512'
	* hash for 'elastic-agent-7.12.1-windows-x86_64.zip' not found
08:49:24.804
warn
elastic_agent
C:\Program Files\Elastic\Agent\data\elastic-agent-9b2fec\logs\elastic-agent-json.log
[elastic_agent][warn] Elastic Agent status changed to: 'degraded'
08:49:24.804
info
elastic_agent
C:\Program Files\Elastic\Agent\data\elastic-agent-9b2fec\logs\elastic-agent-json.log
[elastic_agent][info] Elastic Agent status changed to: 'online'