# Redundant syslog processing

**URL:** <https://discuss.elastic.co/t/redundant-syslog-processing/363178>\
**Category:** Elastic Agent\
**Created:** [July 15, 2024, 6:01pm UTC](https://discuss.elastic.co/t/redundant-syslog-processing/363178 "2024-07-15T18:01:08Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dmgeurts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmgeurts/32/95115_2.png) [@dmgeurts](https://discuss.elastic.co/u/dmgeurts)\
**Post date:** [July 15, 2024, 6:01pm UTC](https://discuss.elastic.co/t/redundant-syslog-processing/363178/1 "2024-07-15T18:01:08Z")

</div>

I'm testing log collection from some Palo Alto firewalls and am considering what the best design might be for reliable and redundant log collection.

What do other do?

- Log to two collectors and have Elastic deduplicate the entries?
- Run two log collectors on the same IP address.
  - Anycast / load-balanced / Heartbeat / Other?

- Accept that is a log collector is down for maintenance, there are no logs.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 15, 2024, 6:37pm UTC](https://discuss.elastic.co/t/redundant-syslog-processing/363178/2 "2024-07-15T18:37:08Z")

</div>

Configure your firewall device to send logs to a load balancer, then behind this load balancer you will put 2 or more log collectors like Elastic Agent or Logstash for example.

If you use a load balancer like nginx or haproxy and also want redundancy for it you could have 2 or more servers running it and use a virtual ip address with keepalived to change the ip address if some of the servers go down.

---

<div class="post-metadata">

**Author:** ![dmgeurts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmgeurts/32/95115_2.png) [@dmgeurts](https://discuss.elastic.co/u/dmgeurts)\
**Post date:** [July 19, 2024, 6:58pm UTC](https://discuss.elastic.co/t/redundant-syslog-processing/363178/3 "2024-07-19T18:58:29Z")

</div>

FYI, it did make me wonder if there's value in using 'assured' transmission. In the end, I went for plain and simple HA and round-robin load-balancing.

The reason for considering this was that HAProxy now supports a ring buffer, but a ring can only have one server. So to load-balance one would have to buffer to a loopback interface and then load-balance from there to two or more servers. Or vice versa (LB \>\> to multiple local frontends each with a with ring buffer and one server. Which sounds rather inefficient to me. This all gets confusing quickly, and I think one is better off reducing timers for health checks and fail-over.

Alternatively, Rsyslog can do reliable transmission with some buffering, but then I fail to see the point of combining HAProxy with Rsyslog, why not do it all without HAProxy then?

I switched to TCP transmission both from the sender and when forwarding to the collector, as then HAProxy can tell if a server is no longer available. UDP after all is fire and forget.