Elastic Agent + Proxy + Fleet Server in Cloud not ingesting logs

Elastic Security
,
1

Hello,

I'm facing some issues while trying to use Endpoint Security by installing Elastic Agent behind a proxy.

Environment details:

Elasticsearch and Fleet server in Cloud v8.1.
Endpoints without access to Internet forward traffic through Proxy.
Collect agent logs and metrics are enabled on the policy

image
image774×155 15.2 KB

Nginx Proxy

The agent enrolls successfully to Fleet, but after some moment becomes in UNHEALTHY state.

Happens the same behavior as described on the following discussion.

.\elastic-agent.exe enroll --url=https://{{ PROXY_URL }}:{{ PORT }} --enrollment-token={{ TOKEN }}

Squid Proxy (testing purposes)

Agent enrolls successfully to Fleet and stays in HEALTHY state, but there are no logs ingested.

.\elastic-agent.exe enroll --url=https://{{ FLEET_CLOUD_URL }}:{{ PORT }} --enrollment-token={{ TOKEN }} --proxy-url=http://{{ PROXY_URL }}:{{ PORT }} --insecure

Squid Logs

1651220452.529   1026 172.31.140.243 TCP_TUNNEL_ABORTED/200 7010 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651220578.577  91304 172.31.140.243 TCP_TUNNEL/200 9164 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651220992.213 503628 172.31.140.243 TCP_TUNNEL/200 6481 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651222628.005    883 172.31.140.243 TCP_TUNNEL_ABORTED/200 7010 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651228152.442    757 172.31.140.243 TCP_TUNNEL_ABORTED/200 7010 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651228255.491  91114 172.31.140.243 TCP_TUNNEL/200 9169 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651228551.764 386263 172.31.140.243 TCP_TUNNEL_ABORTED/200 6480 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651229493.539 928887 172.31.140.243 TCP_TUNNEL_ABORTED/200 13548 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651230245.033 724693 172.31.140.243 TCP_TUNNEL/200 6410 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651230588.606 320637 172.31.140.243 TCP_TUNNEL/200 9463 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651231195.713    772 172.31.140.243 TCP_TUNNEL_ABORTED/200 7010 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651231295.272  91565 172.31.140.243 TCP_TUNNEL/200 9144 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651231840.752  54026 172.31.140.243 TCP_TUNNEL_ABORTED/200 6060 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651231840.752 635475 172.31.140.243 TCP_TUNNEL_ABORTED/200 13363 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651231840.752  52708 172.31.140.243 NONE/000 0 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_NONE/- -
1651232228.570 180492 172.31.140.243 NONE/503 0 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_NONE/- -
1651233021.132   1345 172.31.140.243 TCP_TUNNEL_ABORTED/200 7010 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651233122.128  91300 172.31.140.243 TCP_TUNNEL/200 9153 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651233125.590  90644 172.31.140.243 TCP_TUNNEL/200 5989 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651233401.016 368882 172.31.140.243 TCP_TUNNEL/200 6252 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651233696.100 383740 172.31.140.243 TCP_TUNNEL/200 5992 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.123.161.150 -
1651233974.153 366657 172.31.140.243 TCP_TUNNEL/200 5992 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651234270.758 385452 172.31.140.243 TCP_TUNNEL/200 5993 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.123.161.150 -
1651234548.619 366255 172.31.140.243 TCP_TUNNEL/200 6184 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651236476.830  90373 172.31.140.243 TCP_TUNNEL/200 5990 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.123.161.150 -
1651236762.984 375444 172.31.140.243 TCP_TUNNEL/200 5991 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.123.161.150 -
1651237056.291 381932 172.31.140.243 TCP_TUNNEL/200 5992 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651237252.139 284638 172.31.140.243 TCP_TUNNEL_ABORTED/200 5735 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651237378.120  80578 172.31.140.243 TCP_TUNNEL_ABORTED/200 5988 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651237378.120  79930 172.31.140.243 TCP_TUNNEL_ABORTED/200 5735 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.125.130.49 -
1651237925.501  90328 172.31.140.243 TCP_TUNNEL/200 6038 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.123.161.150 -
1651238107.798   7461 172.31.140.243 TCP_TUNNEL_ABORTED/200 7010 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651238142.791    324 172.31.140.243 TCP_TUNNEL/200 5876 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651238143.372    575 172.31.140.243 TCP_TUNNEL/200 5876 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651238143.654    276 172.31.140.243 TCP_TUNNEL/200 5876 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651238143.927    268 172.31.140.243 TCP_TUNNEL/200 5876 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651238205.312  91093 172.31.140.243 TCP_TUNNEL/200 9151 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651238209.978  90454 172.31.140.243 TCP_TUNNEL/200 5989 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -
1651238484.054 368748 172.31.140.243 TCP_TUNNEL/200 6252 CONNECT {{ REDACTED_URL }}:{{ REDACTED_PORT }} - HIER_DIRECT/3.127.228.159 -

By executing the agent manually with verbosity I don't see any error message or anything that could help to understand what's happening. Also, I've tried by setting up different port as the troubleshoot common problems state, but also there was no luck to make it work.

Assumptions

  1. Agent installed in the endpoint can communicate with Fleet Server, as it's able to enroll and keep in healthy state
  2. As per seems a connectivity issue, but not sure in which part and how to remediate it

Questions

  1. There is someone that has experienced similar behavior?
  2. There is ingress connectivity needed from Fleet Server to the endpoint in order to ship logs? If so, any advice on how to implement with a proxy between Elastic Cloud services and multiple endpoints inside private networks?

Any nudge on what I'm missing or how to implement this solution would be more than welcome!

Thanks in advance,

References:

2

Update:

After trying several things, looks like I missed one of the points provided in the documentation to make it work.

Enroll agent with flag --proxy-url + add on the fleet server yml file the variable proxy_url to allow communication between proxy and Elasticsearch.

This topic can be deleted.

Thanks.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.