Endpoint Agent and Proxy Issues

[jad3675 (John Danhires)]

Windows Server 2019
Elastic 8.6

Our environment has some devices that can only communicate with our Elastic cluster via a proxy. The elastic agent is setup with the environment registry keys (HKLM\SYSTEM\CurrentControlSet\Services\Elastic Agent\Environment) for http_proxy and https_proxy. The elastic agent respects and uses these environment variables.

When the server is added to an agent policy that has Endpoint security, the endpoint agent is deployed as expected. However, it is unable to communicate with our cluster because the install is not copying/using the environment variables from the elastic agent registry key.

Should be an easy enough fix - just add the appropriate registry keys to (HKLM\SYSTEM\CurrentControlSet\Services\ElasticEndpoint) and restart the server (since the endpoint service can't be restarted).

After the restart, this error is in the logs - '[elastic_agent.endoint_security] [info] Connecting to server localhost:6789 via HTTP proxy ipv4:(ipaddress of proxy)'.
So the agent is using (and reading ) the proxy config from the registry, it just needs to exclude 'localhost' from being proxied. 'no_proxy=localhost,127.0.0.1' is added to the Environment registry key and the server is restarted.

After the restart, this is in the logs: '[elastic_agent.endoint_security][debug] not using proxy for host in no_proxy list 'dns:///localhost:6789'. So the agent is reading the 'no_proxy' variable and applying it properly.

But....the elastic endpoint agent is not using the proxy config to communicate with fleet or our cluster. The agent logs are littered with communication issues between the endpoint agent, fleet and our cluster. The logs also contain '[elastic_agent.endpoint_security][info] ProxyConfig.cpp Proxy URL:'

Additionally, Process Explorer shows the endpoint-agent is NOT using the proxy, it's showing direct attempted connections to fleet.

Adding http_proxy, https_proxy and no_proxy environmental variables to the system variables and restarting the server did not fix the issue for the endpoint service.

From an admin powershell prompt, running 'endpoint-agent.exe test output' - produces output that shows the proper proxy url and show connectivity to the cluster. If the https_proxy, http_proxy environmental variables are removed, the test output fails since it is attempting a direct connection.
It appears that the executable is respecting the environmental variables and using them. It appears to be an issue with the elasticendpoint service not reading and using the environment variables - unless I am doing something wrong.

Any help would be greatly appreciated.

[gabriel.landau]

Hi @jad3675,

You're correct that Endpoint doesn't support those registry keys. Endpoint protects its service key from modification/tampering, so even an administrator should not be able to set those registry keys.

PS C:\Windows\System32> $environment = [string[]]@(
>>   "HTTPS_PROXY=https://proxy-hostname:proxy-port",
>>   "HTTP_PROXY=http://proxy-hostname:proxy-port"
>>   )
>>
>> Set-ItemProperty "HKLM:SYSTEM\CurrentControlSet\Services\ElasticEndpoint" -Name Environment -Value $environment
>>
Set-ItemProperty : Requested registry access is not allowed.
At line:6 char:1
+ Set-ItemProperty "HKLM:SYSTEM\CurrentControlSet\Services\ElasticEndpo ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : PermissionDenied: (HKEY_LOCAL_MACH...ElasticEndpoint:String) [Set-ItemProperty], SecurityException
    + FullyQualifiedErrorId : System.Security.SecurityException,Microsoft.PowerShell.Commands.SetItemPropertyCommand

PS C:\Windows\System32> reg.exe add HKLM\SYSTEM\CurrentControlSet\Services\ElasticEndpoint /v Anything /t REG_SZ /d "testing"
ERROR: Access is denied.

Endpoint uses the same proxy configuration sent to Agent by Kibana, Starting with 8.7.0, you can set this directly in Kibana following the example here. I summarized the process here:

I hope this helps.

Regards,
Gabriel

[jad3675 (John Danhires)]

Thanks for the quick reply....but it doesn't appear that endpoint supports the environmental variables either as a service? It respects it as a standalone run of the executable.

I was setting the keys in a hacky way - importing the key before I installed the endpoint agent. The installer doesn't clear it out before it install the endpoint-agent.

What is odd, as I pointed out, is that it appears to partially respect the http_proxy variable. The agent couldn't connect to localhost:6789 until I explicitly put in a no_proxy variable in the key.

Unfortunately, only part of our environment needs to use a proxy. We'd have to implement a separate output for these-proxied hosts then, correct?

How does the address the inability of the endpoint agent to download the artifacts (endpoint-exceptionlist-windows-v1, for example) from our single fleet server using a proxy? Would we need to deploy a second fleet server just for the proxied hosts and configure it to use the proxy?

Thanks again!
John

[cmacknz (Craig MacKenzie)]

What specific 8.6 version are you using? In 8.6.2 and 8.7.0 there is a bug in the Elastic Agent where we receive an empty proxy URL from Fleet, and this takes precedence over any proxy you have configured locally. This was an unfortunate side effect of allowing the proxy URL to be configured form the UI. See A Fleet proxy configured with the --proxy-url option when enrolling is not passed to endpoint security · Issue #2304 · elastic/elastic-agent · GitHub

In 8.7.0 the Fleet Proxy URL can be configured from the Fleet UI as described above, setting the proxy to use there before enrolling the agent is one work around if you are affected by this bug.

Another option is to wait for 8.7.1 which will include the fix for this problem. We don't post the release dates publicly but it won't be a long wait.

[jad3675 (John Danhires)]

We're at 8.6.2 - and that's good know with 8.7.1. So if I understand correctly - if I pass the --proxy-url during agent enrollment, that will get passed on to both the elastic and endpoint agent? If so, that's great.

Thanks again! I appreciate the help with this issue.

John

[cmacknz (Craig MacKenzie)]

Yes, the value of --proxy-url set on the command line will be stored and passed to both the Elastic Agent and Endpoint Security agent (with the exception of the bug I mentioned in 8.6.2 and 8.7.0).

[jad3675 (John Danhires)]

Craig -
Thanks again - I upgraded our lab instance to 8.7.1 and verified that the --proxy-url flag is respected with the Elastic and Endpoint agent.

John