Can you put Logstash behind Nginx proxy?

Elastic Stack Logstash
1

Hello,

Can you put Logstash behind Nginx proxy the same way you put Kibana behind an nginx proxy to use a custom domain with SSL? All this while also using HTTP Basic authentication with Nginx?

ElasticSearch, Kibana, and Logstash instances are v7.7 (single server), also runs Nginx with the proxy settings.

Filebeat also 7.7 on a different server that runs nginx where I want to "stash" the logs.

Both servers run Ubuntu.

I can access Kibana, and the elasticsearch Rest api via Nginx proxy just fine, but filebeat from the other server cannot connect to Logstash. (see my next reply)

2

I've tried to put it into a Nginx stream block, but did not work.

stream {
    server {
        listen 443 ssl http2;
        server_name myserverdomain.tld;

        # ssl settings...

        proxy_pass http://127.0.0.1:5044;
    }
}

A filebeat (different server) that monitors Nginx access/error logs is reporting the bellow.

Jul 27 12:42:20 myhost filebeat[24668]: 2021-07-27T12:42:20.948+0300        INFO        [publisher_pipeline_output]        pipeline/output.go:111        Connection to backoff(async(tcp://myserverdomain.tld:443))


Jul 27 12:41:25 myhost filebeat[24668]: 2021-07-27T12:41:25.675+0300        ERROR        [logstash]        logstash/async.go:279        Failed to publish events caused by: lumberjack protocol error

Jul 27 12:40:49 myhost filebeat[24668]: 2021-07-27T12:40:49.485+0300        ERROR        [logstash]        logstash/async.go:279        Failed to publish events caused by: client is not connected
3

That sounds like you are trying to use the HTTP protocol on one side of the proxy and the lumberjack protocol on the other. That is not going to work. You should be able to use a proxy or load balancer if you use HTTP, even with TLS, on both sides. It will then load balance connections.

4

I saw that about the load balancer on Nginx's website, but could not figure it out. Load balancer with just one instance? Is it just called "load balancer" ?

5

If there is only one instance of logstash behind nginx then I think normal usage would be proxy rather than load balancer.

6

Yes, I just cant figure it out. I keep getting that errors.

7

Can anyone shed some light ?

8

Filebeat does not use HTTP, it uses a proprietary protocol over TCP, your proxy configuration is using HTTP, so it won't work, you need to change it.

I do not use nginx, but according to the documentation you just need to remove any mention of http from your configuration. And as you are terminating the SSL connection at the nginx proxy, this is the documentation about SSL.

Try the following configuration:

stream {
    server {
        listen 443 ssl;
        server_name myserverdomain.tld;

        # ssl settings...

        proxy_pass 127.0.0.1:5044;
    }
}
1 Like
9

Hey, thanks for your reply.

So... I'm still getting the errors. Is there a way I can confirm that the server running Filebeat can actually communicate with the server running Logstash? telnet or something?

10

@leandrojmp's reply is partially my solution.

I managed to connect Filebeat to Logstash via Nginx TCP SSL termination, I just couldn't use port 443 (nor 5044). I setup a random port to listen to and it started forwarding with SSL. Probably not possible to use the same port to listen to on Nginx while having the same port used by Logstash (same machine)... maybe configuring logstash to start on a different port and use 5044 as a listening port with Nginx. But this is not a problem for me right now.

Now I need to figure out why Logstash, which sits on the same machine with the Elastic instance, is getting:

Encountered a retryable error. Will Retry with exponential backoff  {:code=>403, :url=>"http://127.0.0.1:9200/_bulk"}

Probably some authentication issue.

I used logstash_system user, maybe it needs a new user not that system reserved one. Who knows..

11

You can not use the same protocol and port twice on a server, so if nginx and logstash are on the same machine you will need different ports.

I would use 5044 for nginx and 50044 for logstash and to avoid confusion I would not use port 443 as port 443 is normally used for https.

Your other error is probably related to authentication, but I would recommend that you create another post on the forum if you can not fix it.

1 Like
12

Yes, already figured everything and I'm getting logs on my Elastic instance :smiley:
Thanks!

13

I'm leaving here the Nginx configuration I used that I figured out with the help of given answers.

First you'll need the latest NGINX Open Source compiled with the --with-stream and with-stream_ssl_module configuration parameters

Check your system with:

nginx -V 2>&1 | tr ' ' '\n' | grep stream

Next, inside /etc/nginx edit the nginx.conf file and append to the bottom:

include /etc/nginx/streams-enabled/*;

Create that folder if it doesn't exist

$ sudo mkdir /etc/nginx/streams-enabled

Create a file inside sites-available and name it what ever you want. eg. logstash.proxy

Edit the file and add the following basic configuration

stream {
	upstream logstash {
		server 127.0.0.1:5044;
	}

	server {
		listen 5544 ssl;
		proxy_pass logstash;
		
		# SSL
        ssl_certificate /etc/letsencrypt/live/YOURDOMAIN/fullchain.pem;
	    ssl_certificate_key /etc/letsencrypt/live/YOURDOMAIN/privkey.pem;
    }
}

Symlink that file into the streams-enabled folder

$ sudo ln -s /etc/nginx/sites-enabled/logstash.proxy /etc/nginx/streams-enabled/logstash-proxy

Test nginx with the nginx -t command. If everything's fine, you should be able to restart Nginx with this configuration and your beats will be able to communicate with your Logstash instance.

There are more steps to do with Logstash and *beats configurations, but the above should cover the Nginx part.

Please be careful with the above and don't use it in Production. It is only meant for education and fiddling around.

if you want to use it on a publicly accessible system (DigitalOcean, Linode, Whatever...), you should have a firewall up and only allow the machines running the beats through.

// Ubuntu with ufw
$ sudo ufw allow from your.beats.ip.address to any port 5544 proto tcp // or any port you used above
$ sudo ufw reload
$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----         
5544/tcp                   ALLOW       your.beats.ip.address
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.