Load balancing data from Filebeat to Logstash using Nginx

Elastic Stack Logstash
1

Hello,

I have ELK cluster with 3 nodes for elasticsearch(master+data) , 2 nodes for logstash(Active and Passive) and 2 for kibana.I have to utilize both the logstash nodes.

I am using filebeat to parse logs and send data to logstash.
Since there are multiple sources, I am thinking of using nginx as a load balancer.(filebeat -> nginx -> logstash1 and logstash2).

I know filebeat can do load balancing and it cannot send output data to nginx, but i want to try it using nginx. Can this be done?
I have tried this but getting "ERR Failed to publish events caused by: lumberjack protocol error" in filebeat. I have setup the logstash upstream servers in nginx.

Thanks!

2

Hi @Nikhil04,

It might work if you use nginx in TCP/streaming mode... But why? As you said, Filebeat can loadbalance over several Logstash instances and Filebeat can back off if Logstash is not keeping up. You probably loose that going through nginx

3

Hi @A_B,

I am just trying this out to see which load balancing method (filebeat or nginx) will better server my purpose.As you may know, nginx also has more additional features and I would like to test them here.

But first I need to make it work using nginx as LB, I have tried using nginx in TCP/stream mode but it is giving error that it cannot publish events "ERR Failed to publish events caused by: lumberjack protocol error", I am giving IP and port of nginx in filebeat output.

Regards,
Nikhil

4

After a quick web search it looks like Filebeat is ~not~ using TCP. The actual network protocol is lumberjack protocol. This means nginx does not know how to handle it.

I could be completely wrong as I spent all of 2 minutes researching it...

5

Spent 2 more minutes looking at it as I got curious...

From Beats protocol

The lumberjack protocol sits on TCP. With TLS support you have one of TCP/lumberjack or TCP/TLS/lumberjack.
6

Hi @A_B

Thanks for the update.
So do you mean nginx as load balancer sitting between logstash and filebeat should work?

7

No I don't think it can work. But I could be wrong. I have not tested it and have no plans of doing it either so this is partially speculation on my part though :smiley:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

10

You can use Nginx as the load balancer for logstash but as @A_B mentioned, you need to do this on TCP level because filebeat uses lumberjack to communicate with logstash which is a protocol that sits on the top of TCP. For TCP load-balancing in Nginx you need to use the stream block instead of the http block. Your configuration should look something like this:

stream {
    upstream logstash {
        server logstash-1:5044;
        server logstash-2:5044;
    }

    server {
        listen       5044;
        listen  [::]:5044;

        proxy_pass logstash;
    }
}

Here I used Docker to create multiple logstash containers that each one uses port 5044. Feel free to change it based on your setup.

11

Since I don't think its been mentioned in this post yet... some drawbacks that should be made clear:

  • Logging often makes use of the source IP (this is particularly true of LAN-based logging protocols such as syslog; less so of logging-as-a-service). Reverse-proxying this Beats traffic will cause Logstash (beats input) to see all traffic as coming from Nginx
  • You won't be able to use SSL/TLS client certificates...
  • This is very unlikely to be the happy path with regard to tools like Fleet and the Elastic Agent, if that is of interest.
  • An alternative design, often desired with TCP reverse-proxying in general, is to use the PROXY protocol, which means that both Nginx (or HAProxy, where this came from) and the backend input (beats input) needs to support PROXY protocol... but that functionality is not implemented yet in the stock Logstash inputs.

Alternative designs include:

  • using floating IPs (eg. Pacemaker) to float various dedicated service IPs around a cluster a cluster [while not really necessary for Beats, this does prove very useful for other types of logging endpoints, such as for Syslog.
  • using DNS Round Robin (really not recommended as a lot of things don't tend to lookup the DNS name very often, particularly network equipment).
  • GSLB (see DNS Round Robin, just run faster, unless your head is already firmly stuck in the clouds, in which case it could well be your best friend.)
  • layer-4 load-balancing, which is effectively playing tricks with NAT.

Designs to avoid:

  • anything where you point clients at Elasticsearch directly [this is a concern regarding trust boundaries]
  • anything where you point clients at Kafka directly [concern regarding trust, versioning and robustness]

I would strongly urge you to ensure that you have some sort of persistent queue, either what Logstash provides, or using tools like Kafka (which bring other architectural opportunities).

My own experience tells me that Pacemaker is really quite useful for floating some IPs around a small cluster; I use that to surface logstash (beats), as well as various rsyslogd instances. All of these put their data into Kafka, which then gets consumed by another logstash which does the enrichment and passed it into Elasticsearch. This will very likely be too big and complex for small deployments, but gives a lot of flexibility and control, particularly if you want a place to filter data before sending it into the likes of Elastic Service. Elasticsearch indexing should be your slowest part of your pipeline, so I wouldn't be concerned about scaling out Logstash for performance reasons (just know how not not make regular expressions that suck).

PS. Also worth noting is that 'Beats' is being renamed to 'Elastic Agent'... no idea what implications that would have protocol wise.

If you haven't already, check your thinking against Deploying and Scaling Logstash | Logstash Reference [7.14] | Elastic