Redundant syslog processing

Elastic Stack Elastic Agent
1

I'm testing log collection from some Palo Alto firewalls and am considering what the best design might be for reliable and redundant log collection.

What do other do?

  • Log to two collectors and have Elastic deduplicate the entries?
  • Run two log collectors on the same IP address.
    • Anycast / load-balanced / Heartbeat / Other?
  • Accept that is a log collector is down for maintenance, there are no logs.
2

Configure your firewall device to send logs to a load balancer, then behind this load balancer you will put 2 or more log collectors like Elastic Agent or Logstash for example.

If you use a load balancer like nginx or haproxy and also want redundancy for it you could have 2 or more servers running it and use a virtual ip address with keepalived to change the ip address if some of the servers go down.

1 Like
3

FYI, it did make me wonder if there's value in using 'assured' transmission. In the end, I went for plain and simple HA and round-robin load-balancing.

The reason for considering this was that HAProxy now supports a ring buffer, but a ring can only have one server. So to load-balance one would have to buffer to a loopback interface and then load-balance from there to two or more servers. Or vice versa (LB >> to multiple local frontends each with a with ring buffer and one server. Which sounds rather inefficient to me. This all gets confusing quickly, and I think one is better off reducing timers for health checks and fail-over.

Alternatively, Rsyslog can do reliable transmission with some buffering, but then I fail to see the point of combining HAProxy with Rsyslog, why not do it all without HAProxy then?

I switched to TCP transmission both from the sender and when forwarding to the collector, as then HAProxy can tell if a server is no longer available. UDP after all is fire and forget.