Endpoint service not honoring proxy environment variables

We're testing out deploying Defend to our fleet but are running into an issue where the endpoint service isn't honoring the HTTP_PROXY or HTTPS_PROXY environment variables, which for us is needed to push documents into Elasticsearch and for connecting to the artifacts repo. We use the environment variables rather than different Fleet policies due to the large number of differing environments/proxy URLs we need to configure.

For the elastic-agent service we use a systemd drop-in file to set the environment variables to our proxy which works well, but the same applied to the ElasticEndpoint service doesn't.

$ systemctl show ElasticEndpoint | grep _PROXY
Environment=HTTP_PROXY=<proxy url> HTTPS_PROXY=<proxy url> 

I've verified doing a manual curl without the proxy environment variables times out, but with the proxy environment variables set works as expected. In the endpoint logs I can see it trying to connect to the security artifacts repo but it hitting the timeout.

{"@timestamp":"2023-03-07T22:44:50.81539563Z","agent":{"id":"13bf9c3e-0efb-4c64-b604-97a6992c81c6","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":1814,"name":"Artifacts.cpp"}}},"message":"Artifacts.cpp:1814 Checking if new global artifacts manifest is available","process":{"pid":6701,"thread":{"id":6731}}}
{"@timestamp":"2023-03-07T22:47:01.695039657Z","agent":{"id":"13bf9c3e-0efb-4c64-b604-97a6992c81c6","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":327,"name":"Http.cpp"}}},"message":"Http.cpp:327 CURL error 28: Timeout was reached [Failed to connect to artifacts.security.elastic.co port 443 after 130875 ms: Connection timed out]","process":{"pid":6701,"thread":{"id":6731}}}

Is there another way I can set the proxy environment variables for the service to use that I might be missing? I couldn't find a repo for this code to dig deeper either so I'm assuming it's closed-source.

Hi @indyg,

Thanks for testing out Defend. Unfortunately, for the endpoint portion of defend, the only way to configure proxies currently is to use the fleet policies option.

This response took a little longer than I wanted to put together because it didn't quite seem like everything was working as it should and I wanted to figure out why.

In the end, we expect agent and endpoint to use the same proxy settings. However, environment variable proxy settings are not propagated from Agent to Endpoint. I've filed a bug here for tracking the issue.

Let me know if you have any other questions about it.

Thanks,
-Nick Fritts

@indyg

I looked into this more.

There are 2 things that I think are a bug here. The first is the agent bug. You shouldn't have to use the systemd drop in for endpoint. If you used it on Agent, it should propagate to Endpoint.

The second is that for one of the artifacts lookups, it's not setting the proxy properly regardless of how it's set. I opened an issue here so you can track when it is fixed.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.