Suppose I have several services. They are all configured to send their logs to ES.
In complex systems it is quite usual that the same messages (sometimes even errors) are logged every day. I was to create a dashboard where only abnormal messages are showed. They would indicate that something unusual happens.
The first thing to come in mind is just to set a filter and filter out all "regular" messages. In some systems it would be as easy as filter out INFO messages, but, as I said, it can be hard in complex systems. Moreover, some enterprise services do not provide log level in their logs!
The second thought is to filter messages by text. But it will be a huge filter and it would be hard to create and maintain it.
Another idea is to perform an es aggregation on log write (or lazily) to find similar messages. But I'm not quite sure how to do it and it sound slow.
So, what is the common approach to show only abnormal messages? Any ideas would be appreciated.