Observability solutions use to work in a different way. They collect the data as soon as it is available in the edge systems and send them to a central point. With beats you can send this data (logs, metrics, traces...) easily to Elasticsearch. Once in Elasticsearch they can be visualized or analyzed in any way. This has the single disadvantage that you have all the data in Elasticsearch, but this has many advantages:
- When having an infrastructure with many systems It is a good practice to send the logs out of the machines as soon as possible, so if the machine is lost or compromised in any way the logs are still available in a central point of trust.
- Previous point is double important if you use containers or virtual machines. When you delete them you may lose their logs if you haven't send them to a central point.
- You don't have to worry about keeping your logs in your systems for a long time so they can be later read.
- Having all the data in a central point allows you to make queries and aggregations much faster than having to read them system by system. Imagine for example in your use case that your users want to have a visualization to view on how many systems an error is happening, or with what frecuency, this is easy to do if you can search along all the logs of all your machines from a central point. But is complex to do if you have to query system by system and then somehow aggregate the data.
- Logs lifecycle can be decided from a central point. E.g. you can decide for how long you want to keep your data. If you want to keep logs for a longer time, you don't depend on the capacity of each one of your systems, you can scale Elasticsearch as you go.
- You can extract information as timestamps, hostnames, requests done... from your logs.
If you want to give a try to this approach you only need to install Filebeat in some of your systems, configured with Elasticsearch as its output. Then you can use for example the Kibana Logs UI to visualize the logs from a centra point. Take a look to https://www.elastic.co/log-monitoring