Enable filebeat and logstash caching during unavailablity

Elastic Stack Logstash
1

Hello,

I hope you and your loved ones are safe and healthy.

I am running a cluster that collects logs from sources on the internet. I need to enable caching of logs in case the next hop is not reachable as dropping logs is detrimental to my project.
Following is my architecture:

image
image1490×954 69.8 KB

  1. Various log sources (mostly running Linux) send logs using Filebeat to my homelab which are collected by Logstash.
    A. This is where the first unavailability can occur. As I use home ISP and do not commercial agreement, there are availability issues. How do I enable caching of logs (up to 48 hours) at filebeat in case the next hop (logstash hosted in my homelab) is not available.

I have enabled deduplication in filebeat + logstash using: Deduplicate data | Filebeat Reference [8.3] | Elastic

  1. Logstash sends logs to two Elasticsearc nodes, while I have a solid (wired) connectivity in the homelab if I want logstash to collect and store logs if ES nodes are down, how do I enable that?
    My logstash instance is running on an RPi 4 (2 GB RAM) and my logs events per second (EPS) is around 700, the storage on the RPi is 128 GB with a NAS volume mounted that has 10 TB

Thank you very much.

2

Maybe insert kafka between filebeat and logstash, and add persistent queues in logstash with enough disk space to handle the longest possible elasticsearch outage.

1 Like
3

Yes that could be one way:

  1. I could host Kafka on a seperate cloud service provider (CSP)to cache the logs. Making availablity a cloud CSP ownership. However,
  2. instead of the additional cost is there no way to increase caching at Filebeat?
    As in if Filebeat cannot reach Logstash for up to 48 hours it keeps the logs? From the linux configuration perspective the logs are on the disk until 7 days and I can easily keep it for 30 days.
4

A question about increasing caching in filebeat should be in the filebeat forum, not the logstash forum.

5

Sure, I'll repost it there and keep this one for Logstash? Thank you very much.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.