# How to get ML Categorization results break down by users **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [December 1, 2022, 6:43pm UTC](https://discuss.elastic.co/t/how-to-get-ml-categorization-results-break-down-by-users/320283 "2022-12-01T18:43:12Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![CamillePapillon](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@CamillePapillon](https://discuss.elastic.co/u/CamillePapillon)\ **Post date:** [December 1, 2022, 6:43pm UTC](https://discuss.elastic.co/t/how-to-get-ml-categorization-results-break-down-by-users/320283/1 "2022-12-01T18:43:12Z")
Hi! I have an input index with tons of different logs messages coming from many different devices (same product type = similar logs, different users). My source index is break down with : timestamp, serial\_number, message\_data. When I do the ML categorization, I have 1536 different mlcategories across all message\_data. What I would like to achieve is to have the num\_matches per mlcategory per serial\_number for a specific time range. What I have right now as output is num\_matches per mlcategory, but over all serial\_number. I can't figure our how to break it down by serial\_number. To sum up, this is what I want to achieve: 1. Build a categorization ML job on message\_data. With that, I have approx. 1500 mlcategory. So each message\_data can be mapped to one of the 1500 grok pattern (each mlcategory = grok pattern). 2. I want the total count of mlcategory (the grok pattern) for each SN per time bucket, instead of having the count from all SN. 3. I want to have after the total count over specific time range. My data would look like that : ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b4ef3f31c283d60b722067795a0351dd1749a864.png) Note: I also do not have log format documentation for these devices and are hoping ML categorization can help us. A solution that was recommended was be a pipeline to pipeline communication using the generated grok patterns from the categorization job. Of course, the grok patterns are a bit too much and can add to processing time of every log that goes through ingest pipeline. ---
**Author:** ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)\ **Post date:** [December 1, 2022, 7:03pm UTC](https://discuss.elastic.co/t/how-to-get-ml-categorization-results-break-down-by-users/320283/2 "2022-12-01T19:03:16Z")
In your `detector` you need to set `partition_field_name` to `serial_number` and `by_field_name` to `mlcategory`. So for example you might [create the job](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html) like this: ```auto PUT _ml/anomaly_detectors/categories_per_serial_number?pretty { "analysis_config": { "bucket_span": "15m", "categorization_field_name": "15m", "detectors": [ { "detector_description": "Count of category partitioned on serial number", "function": "count", "by_field_name": "mlcategory", "partition_field_name": "serial_number" } ], "influencers": ["serial_number", "mlcategory"] }, "data_description": { "time_field": "@timestamp", "time_format": "epoch_ms" }, "analysis_limits": { "model_memory_limit": "1gb" }, "results_index_name": "cat-by-sn" } ``` Depending on which UI wizard you're using you might need to "convert to an advanced job" and edit the JSON directly to achieve this. Or of course you can create the job using the [Elasticsearch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html) directly. ---
**Author:** ![CamillePapillon](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@CamillePapillon](https://discuss.elastic.co/u/CamillePapillon)\ **Post date:** [December 1, 2022, 7:21pm UTC](https://discuss.elastic.co/t/how-to-get-ml-categorization-results-break-down-by-users/320283/3 "2022-12-01T19:21:30Z")
Hi ! Thanks for your answer. But if I do that, each serial\_number will have independant categories (no mlcategories in common). I tried it and I was left with 3M different categories ( 1500 cat x 2000 different SN) What I would like is to create a "universal" categorization amongst all serial\_numbers messages (because serial\_numbers have similar log messages since it comes from the same product) , and then after have the count of occurence of each mlcategories for each serial\_number. ---
**Author:** ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)\ **Post date:** [December 1, 2022, 7:54pm UTC](https://discuss.elastic.co/t/how-to-get-ml-categorization-results-break-down-by-users/320283/4 "2022-12-01T19:54:14Z")
> [@CamillePapillon](#): > > What I would like is to create a "universal" categorization amongst all serial\_numbers messages To do this you need to make sure `per_partition_categorization.enabled` is `false`. Search for `per_partition_categorization` in [Create anomaly detection jobs API | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html) for details. But `false` is the default so it’s surprising you’re getting 3 million categories if you didn’t know about that setting. ---
**Author:** ![CamillePapillon](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@CamillePapillon](https://discuss.elastic.co/u/CamillePapillon)\ **Post date:** [December 6, 2022, 7:26pm UTC](https://discuss.elastic.co/t/how-to-get-ml-categorization-results-break-down-by-users/320283/5 "2022-12-06T19:26:09Z")
Thank you very much ! I did the ML job and it works. However, I see that some categories ( from result\_type = category\_definition) are not present in the record results (result\_type = record). Let's say that I take mlcategory=2, in category\_definition I see the regex and num\_matches = 5M. When I look for a document with result\_type = record and mlcategory =2 , I have no results. I should have at least 1 since I have 5M matches with that regex in my data. My question is, is it normal to have categories defined in category definition but not present in the records ? Does the record results holds all the results or I have to look somewhere else to have the number of occurence of mlcategory per serial\_numbers? Thanks ---
**Author:** ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)\ **Post date:** [December 6, 2022, 8:08pm UTC](https://discuss.elastic.co/t/how-to-get-ml-categorization-results-break-down-by-users/320283/6 "2022-12-06T20:08:52Z")
> [@CamillePapillon](#): > > My question is, is it normal to have categories defined in category definition but not present in the records ? Yes, this is normal. Records are only created when there's an anomaly. > [@CamillePapillon](#): > > Does the record results holds all the results or I have to look somewhere else to have the number of occurence of mlcategory per serial\_numbers? Anomaly detection jobs are for detecting anomalies. It sounds like you want counts per category per time bucket even when there are no anomalies. You might be able to do this using an aggregation that has a `categorize_text` aggregation to create the categories, a `date_histogram` aggregation to split into time buckets and a `terms` aggregation to split by `serial_number`. Then the `doc_count` at the lowest level would be the numbers you want. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [December 21, 2022, 6:09pm UTC](https://discuss.elastic.co/t/how-to-get-ml-categorization-results-break-down-by-users/320283/8 "2022-12-21T18:09:10Z")
Try something like: ```auto POST message-logs-*/_search { "size": 0, "aggs" : { "daily" : { "date_histogram": { "field": "timestamp", "fixed_interval": "1d" }, "aggs": { "categories": { "categorize_text": { "field": "message_data", "similarity_threshold" : 20 }, "aggs": { "by_sn": { "terms" : { "field": "serial_number.keyword" } } } } } } } } ``` ---
**Author:** ![CamillePapillon](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@CamillePapillon](https://discuss.elastic.co/u/CamillePapillon)\ **Post date:** [January 10, 2023, 7:57pm UTC](https://discuss.elastic.co/t/how-to-get-ml-categorization-results-break-down-by-users/320283/10 "2023-01-10T19:57:52Z")
Thanks ! But it is still not giving me any results... with this query : ```auto POST message-logs-*/_search { "size":0, "aggs" : { "daily" : { "date_histogram": { "field": "@timestamp", "fixed_interval": "1d" }, "aggs": { "categories": { "categorize_text": { "field": "message_data", "similarity_threshold" : 20 }, "aggs": { "by_sn": { "terms" : { "field": "serial_number.keyword" } } } } } } } } ``` I have this output : ```auto { "took" : 3, "timed_out" : false, "_shards" : { "total" : 0, "successful" : 0, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 0, "relation" : "eq" }, "max_score" : 0.0, "hits" : [] } } ``` ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [January 12, 2023, 12:45pm UTC](https://discuss.elastic.co/t/how-to-get-ml-categorization-results-break-down-by-users/320283/11 "2023-01-12T12:45:33Z")
The query in my last post works on an index that I have in my cluster, so I know that the syntax is not incorrect. There must be something simple that is incorrect in your setup (i.e. is your time field not really `@timestamp` but something else, like `timestamp`?) Figure out where the problem is by simplifying the query. First do just the `date_histogram` part: ```auto POST message-logs-*/_search { "size": 0, "aggs": { "daily": { "date_histogram": { "field": "@timestamp", "fixed_interval": "1d" } } } } ``` If that works, then add the sub aggregation of `categorize_text`: ```auto POST message-logs-*/_search { "size": 0, "aggs" : { "daily" : { "date_histogram": { "field": "@timestamp", "fixed_interval": "1d" }, "aggs": { "categories": { "categorize_text": { "field": "message_data", "similarity_threshold" : 20 } } } } } } ``` ...and see if that works, and so on. You'll find out which part is not working for you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 9, 2023, 12:46pm UTC](https://discuss.elastic.co/t/how-to-get-ml-categorization-results-break-down-by-users/320283/12 "2023-02-09T12:46:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.