# Aggregate similar logs in kibana **URL:** \ **Category:** Kibana\ **Created:** [August 5, 2016, 8:55am UTC](https://discuss.elastic.co/t/aggregate-similar-logs-in-kibana/57322 "2016-08-05T08:55:54Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Chelovek\_Chelovechni](https://avatars.discourse-cdn.com/v4/letter/c/9d8465/32.png) [@Chelovek\_Chelovechni](https://discuss.elastic.co/u/Chelovek_Chelovechni)\ **Post date:** [August 5, 2016, 8:55am UTC](https://discuss.elastic.co/t/aggregate-similar-logs-in-kibana/57322/1 "2016-08-05T08:55:54Z")
Suppose I have several services. They are all configured to send their logs to ES. In complex systems it is quite usual that the same messages (sometimes even errors) are logged every day. I was to create a dashboard where only abnormal messages are showed. They would indicate that something unusual happens. The first thing to come in mind is just to set a filter and filter out all "regular" messages. In some systems it would be as easy as filter out INFO messages, but, as I said, it can be hard in complex systems. Moreover, some enterprise services do not provide log level in their logs! The second thought is to filter messages by text. But it will be a huge filter and it would be hard to create and maintain it. Another idea is to perform an es aggregation on log write (or lazily) to find similar messages. But I'm not quite sure how to do it and it sound slow. So, what is the common approach to show only abnormal messages? Any ideas would be appreciated. ---
**Author:** ![mbagatini](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mbagatini](https://discuss.elastic.co/u/mbagatini)\ **Post date:** [August 5, 2016, 11:24am UTC](https://discuss.elastic.co/t/aggregate-similar-logs-in-kibana/57322/2 "2016-08-05T11:24:42Z")
Maybe you could use a grok filter on logstash to capture the NORMAL messages, and do nothing to them. Then all the ones that are not normal could be tagged (using a field for that) appropriately, so on Kibana you would just have to filter by that field. ---
**Author:** ![Chelovek\_Chelovechni](https://avatars.discourse-cdn.com/v4/letter/c/9d8465/32.png) [@Chelovek\_Chelovechni](https://discuss.elastic.co/u/Chelovek_Chelovechni)\ **Post date:** [August 5, 2016, 12:48pm UTC](https://discuss.elastic.co/t/aggregate-similar-logs-in-kibana/57322/3 "2016-08-05T12:48:14Z")
It seems that it would be hard to create and maintain this filter. ---
**Author:** ![mbagatini](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mbagatini](https://discuss.elastic.co/u/mbagatini)\ **Post date:** [August 5, 2016, 1:02pm UTC](https://discuss.elastic.co/t/aggregate-similar-logs-in-kibana/57322/4 "2016-08-05T13:02:06Z")
I don't think there is a built-in feature that detects patterns and deviations in strings coming from different sources and in assorted formats. You probably will not be able to avoid having to write and maintain something to sort that. ---
**Author:** ![mbagatini](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mbagatini](https://discuss.elastic.co/u/mbagatini)\ **Post date:** [August 5, 2016, 1:03pm UTC](https://discuss.elastic.co/t/aggregate-similar-logs-in-kibana/57322/5 "2016-08-05T13:03:09Z")
If your "normal" strings are really regular, you could try to sort them out by size I guess. ---
**Author:** ![Chelovek\_Chelovechni](https://avatars.discourse-cdn.com/v4/letter/c/9d8465/32.png) [@Chelovek\_Chelovechni](https://discuss.elastic.co/u/Chelovek_Chelovechni)\ **Post date:** [August 5, 2016, 1:09pm UTC](https://discuss.elastic.co/t/aggregate-similar-logs-in-kibana/57322/6 "2016-08-05T13:09:29Z")
> [@mbagatini](#): > > If your "normal" strings are really regular, you could try to sort them out by size I guess. You mean that errors are usually longer? > [@mbagatini](#): > > I don't think there is a built-in feature that detects patterns and deviations in strings coming from different sources and in assorted formats. What is there is single format of logs? Would it help? ---
**Author:** ![mbagatini](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mbagatini](https://discuss.elastic.co/u/mbagatini)\ **Post date:** [August 5, 2016, 1:18pm UTC](https://discuss.elastic.co/t/aggregate-similar-logs-in-kibana/57322/7 "2016-08-05T13:18:45Z")
I mean the errors could be different. If all of your normal messages have the same size X you could look for the ones which size is not X. Usually in log files when there is an error the stack trace is logged along, making error messages bigger. I can't really help you more without knowing your use case, I'm just trying to give you some ideas. If you have a single format for the logs, why is it so difficult to have a filter on logstash for that? ---
**Author:** ![Chelovek\_Chelovechni](https://avatars.discourse-cdn.com/v4/letter/c/9d8465/32.png) [@Chelovek\_Chelovechni](https://discuss.elastic.co/u/Chelovek_Chelovechni)\ **Post date:** [August 5, 2016, 1:36pm UTC](https://discuss.elastic.co/t/aggregate-similar-logs-in-kibana/57322/8 "2016-08-05T13:36:00Z")
Filtering by size indeed seems reasonable. Some abstract example of logs ``` opening session id xxxx with params bla bla bla //normal performing request form to with bla bla bla //normal sending response from to with bla bla bla //normal closing session id xxxx with params bla bla bla //normal opening session id xxxx with params bla bla bla //normal performing request form to with bla bla bla //normal sending response from to with bla bla bla //normal closing session id xxxx with params bla bla bla //normal flushing some buffers by request //!!!strange!!! opening session id xxxx with params bla bla bla //normal performing request form to with bla bla bla //normal sending response 500 from page with well known bug //normal closing session id xxxx with params bla bla bla //normal opening session id xxxx with params bla bla bla //normal performing request form to with bla bla bla //normal sending response 500 from uxepected page //!!!strange!!! closing session id xxxx with params bla bla bla //normal ``` Filtering can be done, yes, but one need to maintain a large set of filters or even write a separate app to decide what is normal, what is not. It is a dull work. It would be nice to get some piece of logic which just decides "oh, I've seen something similar before" or "unusual message, haven't seen anything similar in weeks". ---
**Author:** ![mbagatini](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mbagatini](https://discuss.elastic.co/u/mbagatini)\ **Post date:** [August 5, 2016, 1:50pm UTC](https://discuss.elastic.co/t/aggregate-similar-logs-in-kibana/57322/9 "2016-08-05T13:50:05Z")
Other than the size you could try have all the messages as analysed strings in the index, and then do some kind of aggregation to identify which ones have words differing from the average. I don't even know if this is possible, but it's another idea. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:41pm UTC](https://discuss.elastic.co/t/aggregate-similar-logs-in-kibana/57322/10 "2017-07-06T13:41:52Z")