Hello,
It seems that there is a bug in the ml display in Kibana ( Categorization> rare occurrences job)
you can see below red squares, but actually there aren't any major anomalies
Hi @liorg2!
What you're seeing is a representation of how we aggregate scores into an overall score, and how we aggregate scores in time.
Firstly, the top level swim lane marked "Overall" gives you a score based on how unusual the all of the events are within that time period. So, in your example, the red tiles are telling you that there are multiple unusual things occurring within that time period. Even if the individual events are not critically unusual on their own, the fact that they are co-occurring is highly unusual. There might not be corresponding colored tiles in the column below a red square because of how we order the influencer swimlanes in the heatmap. The individual category swim lanes are ordered by decreasing rarity and there are presumably many more than 10 categories, which is the default number shown. If you click on a tile in the top-level swim lane it'll resort the category swim lanes and show you all the unusual categories which were present at that time.
Secondly, since you are viewing the results over a period of many months, each individual tile actually contains many buckets as defined in your job configuration. When aggregating scores across time, we take the max score of any bucket in the time frame of the tile. If you were to drill down into a single week, for example, you will likely see much sparser anomalies in the top swim lane.
Finally, you should check that the categories being found are reasonable. How many categories are there in total, do they look plausible, etc. For example, it is possible to tweak categorization by pointing at just a specific field in the data. Note it is also entirely possible to query for the individual category anomalies and alert from these, if you prefer to only know about highly unusual individual categories and not bursts of unusual categories.
Hope that helps! Here's some reference material for scoring and categorization:
3 Likes
It helps a lot, thank you very much
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.