Hello,
My company is currently transitioning from Splunk to Sentinel, despite my preference for Elastic. Sentinel utilizes AMA agents to gather logs from various sources, listening on port 514. However, a significant challenge arises as the Sentinel agent has a fixed buffer size of only 10 GB. Given the volume of logs we handle, this capacity is insufficient, and any network disruption could rapidly fill it.
To address this issue, we are considering incorporating Logstash into our solution. The concept involves using Logstash to ingest logs and then outputting them back to UDP port 514, where an AMA agent will be ready to send the parsed logs to Sentinel. The dilemma lies in Logstash's current limitation, as it only delivers logs to a localhost UDP port that remains active even when the connection to Sentinel is lost.
To tackle this, I propose introducing an additional output, specifically the Microsoft Sentinel Log Analytics Logstash output plugin. This entails creating a generator as the input and a canary table in Sentinel to transmit this data as an output. In the event that the Sentinel website is inaccessible, the Sentinel output will be unable to deliver logs, causing them to accumulate in the persistent queue until the connection to Sentinel is restored. This intricate setup aims to ensure the seamless flow of logs, even during temporary disruptions in the connection to Sentinel.
After this expose I have few questions:
1 - Am I overcomplicating things, and is this solution effective?
2- Is there a more straightforward approach to achieve my objectives, perhaps utilizing a Logstash feature I may be unaware of?
3 - This unconventional idea operates on the assumption that if any logs in the input of a configuration file cannot be delivered to a specific output, none of the logs from that configuration will be delivered. Is this true ?
Thank you very much for your help
