Hello,
We have recently implemented an internal IDS solution. That solution can generate somewhere in the neighborhood of ~30k logs (json format) per minute. Filebeat has a pre-canned module that we are using and it does parse/send the logs as expected. The challenge seems to be keeping up with the volume. During a 10 minute timeframe, there were 282k log entries yet only 41k recorded in Elastic. That lag simply grows with time.
iftop shows no bottlenecks at the interfaces.
top shows no issues with load averages on the cluster (3 node)
The receiving interface is seeing an average of 26k packets per second
What are Filebeat ingest limitations? Other suggestions to troubleshoot?