# Enable filebeat and logstash caching during unavailablity **URL:** \ **Category:** Logstash\ **Created:** [June 29, 2022, 9:23am UTC](https://discuss.elastic.co/t/enable-filebeat-and-logstash-caching-during-unavailablity/308443 "2022-06-29T09:23:55Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Post date:** [June 29, 2022, 9:23am UTC](https://discuss.elastic.co/t/enable-filebeat-and-logstash-caching-during-unavailablity/308443/1 "2022-06-29T09:23:55Z")
Hello, I hope you and your loved ones are safe and healthy. I am running a cluster that collects logs from sources on the internet. I need to enable caching of logs in case the next hop is not reachable as dropping logs is detrimental to my project. Following is my architecture: ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4c5f0b4f17de7a597417dc2c74f51c90faa03e19.png) 1. Various log sources (mostly running Linux) send logs using **Filebeat** to my homelab which are collected by **Logstash**. A. This is where the **first unavailability** can occur. As I use home ISP and do not commercial agreement, there are availability issues. How do I enable caching of logs (up to 48 hours) at filebeat in case the next hop (logstash hosted in my homelab) is not available. _I have enabled deduplication in filebeat + logstash using: [Deduplicate data | Filebeat Reference [8.3] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-deduplication.html)_ 1. **Logstash** sends logs to two **Elasticsearc** nodes, while I have a solid (wired) connectivity in the homelab if I want logstash to collect and store logs if ES nodes are down, how do I enable that? _My logstash instance is running on an RPi 4 (2 GB RAM) and my logs events per second (EPS) is around 700, the storage on the RPi is 128 GB with a NAS volume mounted that has 10 TB_ Thank you very much. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 29, 2022, 4:37pm UTC](https://discuss.elastic.co/t/enable-filebeat-and-logstash-caching-during-unavailablity/308443/2 "2022-06-29T16:37:17Z")
Maybe insert kafka between filebeat and logstash, and add persistent queues in logstash with enough disk space to handle the longest possible elasticsearch outage. ---
**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Post date:** [June 30, 2022, 9:15am UTC](https://discuss.elastic.co/t/enable-filebeat-and-logstash-caching-during-unavailablity/308443/3 "2022-06-30T09:15:15Z")
Yes that could be one way: 1. I could host Kafka on a seperate cloud service provider (CSP)to cache the logs. Making availablity a cloud CSP ownership. **However,** 2. instead of the **additional cost** is there no way to **increase caching at Filebeat?** As in if Filebeat cannot reach Logstash for up to 48 hours it keeps the logs? From the linux configuration perspective the **logs are on the disk until 7 days** and I can **easily keep it for 30 days.** ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 30, 2022, 4:41pm UTC](https://discuss.elastic.co/t/enable-filebeat-and-logstash-caching-during-unavailablity/308443/4 "2022-06-30T16:41:25Z")
A question about increasing caching in filebeat should be in the filebeat forum, not the logstash forum. ---
**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Post date:** [June 30, 2022, 6:25pm UTC](https://discuss.elastic.co/t/enable-filebeat-and-logstash-caching-during-unavailablity/308443/5 "2022-06-30T18:25:36Z")
Sure, I'll repost it there and keep this one for Logstash? Thank you very much. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 28, 2022, 6:25pm UTC](https://discuss.elastic.co/t/enable-filebeat-and-logstash-caching-during-unavailablity/308443/6 "2022-07-28T18:25:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.