Elastic agent -> reverse proxy -> ( fleet server & elasticsearch cluster )

Elastic Stack Elastic Agent
,
13

hello i finally made it work and here are the steps I followed

1- Installed Nginx on a server that can access both the internet and my internal network (I put it in my DMZ)
2- I obtained an SSL certificate for my domain using Let's Encrypt for example or another certificate authority.
(for the sake of this lab I used elasticsearch cert util)

./bin/elasticsearch-certutil cert \
  --name proxy1 \
  --ca-cert /usr/share/elasticsearch/converted_ca/cert.crt \
  --ca-key /usr/share/elasticsearch/converted_ca/private.key \
  --dns proxy1.homelab.lan \
  --ip 192.168.2.36 \
  --pem

3- I created a new Nginx configuration file sudo nano /etc/nginx/sites-available/elastic_reverse_proxy.conf with the following configuration:

#type all your elasticsearch data and master nodes or even better you coordinating_only nodes if you have them nginx will load balance over them  
upstream elasticsearch {
    server 192.168.1.14:9200;
    server 192.168.1.15:9200;
    server 192.168.1.16:9200;
    keepalive 15; #change it according to the number of agents you are deploying using this proxy I guess
}

upstream fleet {
    server 192.168.1.23:8220;  # Fleet server IP and port
    keepalive 15; #change it according to the number of agents you are deploying using this proxy I guess
}

server {
    listen 443 ssl;
    server_name elasticsearch-proxy;

    ssl_certificate /etc/nginx/ssl/proxy1.crt;
    ssl_certificate_key /etc/nginx/ssl/proxy1.key;

    location /api {
        proxy_pass https://fleet;  # Forward to Fleet server

        proxy_http_version 1.1;
        proxy_set_header Connection "Keep-Alive";
        proxy_set_header Proxy-Connection "Keep-Alive";

        proxy_ssl_verify off;  # Disable SSL verification for testing

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location / {
        proxy_pass https://elasticsearch;

        proxy_http_version 1.1;  # Use HTTP/1.1 to enable keep-alive

        proxy_set_header Connection "Keep-Alive";
        proxy_set_header Proxy-Connection "Keep-Alive";

        proxy_ssl_verify off;  # Disable SSL verification

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

4- I enabled the new configuration:

sudo ln -s /etc/nginx/sites-available/fleet-server /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

5- I configured my firewall to allow traffics on ports I used 443 ...

for the sake of this lab let's agree that 192.168.2.36 is my nginx proxy public IP (in production you should update your DNS to point fleet.companya.com to the IP address of your Nginx server...)

6- I then I went to one of my endpoint and installed the elastic agent using:

./elastic-agent install --url=https://192.168.2.36:443 --enrollment-token=my token  --insecure

ofc replace 192.168.2.36:443 with your public IP and the enrollment token as well and use --certificate-authorities=/path/to/ca.crt instead of --insecure in production environments

7- I went to check fleet under kibana UI and found this issue:

Capture d'écran 2024-09-16 140546
Capture d'écran 2024-09-16 1405461840×968 128 KB

Capture d'écran 2024-09-16 140928
Capture d'écran 2024-09-16 140928944×418 32.4 KB

the agent keeps updating and then goes offline without being healthy at all

so I went to check the logs and found the problem:

Capture d'écran 2024-09-17 204323
Capture d'écran 2024-09-17 204323852×775 67.4 KB

thanks to @leandrojmp in Fleet Policy different IP from the server - #3 by leandrojmp

I found that I should add the public IP of my proxy to the output of elasticsearch and fleet server under fleet --> settings like this :

Capture d'écran 2024-09-17 235511
Capture d'écran 2024-09-17 2355111197×746 52.4 KB

I reenrolled the agent and waited for a few minutes (connection through the proxy is quite slow that's one of the biggest down sides of this approach idk if there is a solution for it (if someone knows please suggest a solution))

and here is the result:

Capture d'écran 2024-09-18 000016
Capture d'écran 2024-09-18 000016934×761 51.3 KB

here are some good resources:
Elasticsearch + Kibana behind NGINX reverse proxy with TLS · GitHub /

nginx proxy configuration for elasticsearch · GitHub /

Playing HTTP Tricks with Nginx | Elastic Blog /

for this approach I haven't tried yet I'll certainly do and come with some remarks: Fleet managed Elastic Agent connectivity using a proxy server | Fleet and Elastic Agent Guide [8.15] | Elastic

I hope this help

if you have any remarks or suggestion to improve the performance or security of my approach please feel free to suggest

thank you for taking the time to read !

1 Like