I'm migrating from an on-premises application to a cloud infrastructure using Docker containers. Previously, I wrote my logs to files and used Filebeat to send them to an EventHub. In a containerized environment, this is not feasible, so I am considering using the C# Elastic.Serilog.Sinks.
My question is whether these two solutions can be considered equivalent, or if Filebeat offers greater reliability. What do you think could be an alternative solution to continue using Filebeat? I noticed that I could use blob storage with Filebeat, but this requires specifying the container name. This complicates things because, with horizontal scalability, I can't have multiple dynamic instances writing to the same blob container with the same file name.
If your Elastic Stack is highly available and you can ensure that the Stack can process all incoming logs fast enough to prevent back pressure, you should be okay with using the Sink.
If your Elastic Stack is not highly available, you expect intermittent network errors or you experience bursts of logs that the stack cannot process fast enough, you might be better of with FileBeat.
Alternatives heavily depend on your usecase so I cannot really advise on that. We are using Openshift and are using the Openshift Logging (FluentD / Vector) to ship all logs of all namespaces and pods to the Elastic Stack. Both FluentD and the newer Vector support caching of logs if the Elastic Stack is not able to process the logs right now.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.