# ELK Painless: count unique distinct occurrences **URL:** \ **Category:** Elasticsearch\ **Tags:** painless\ **Created:** [April 17, 2022, 6:49pm UTC](https://discuss.elastic.co/t/elk-painless-count-unique-distinct-occurrences/302582 "2022-04-17T18:49:47Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![angeloimm](https://avatars.discourse-cdn.com/v4/letter/a/d2c977/32.png) [@angeloimm](https://discuss.elastic.co/u/angeloimm)\ **Post date:** [April 17, 2022, 6:49pm UTC](https://discuss.elastic.co/t/elk-painless-count-unique-distinct-occurrences/302582/1 "2022-04-17T18:49:47Z")
Hi there I'm using ELK stack version 7. What I need to do is to count the unique occorence of a value in my indexes. My indexes are created by WSO2 Identity Server version 5.10 and they are so defined: ```auto { "login.wso2.node.ip-2021.03.11" : { "aliases" : { "alias_my_login" : { } }, "mappings" : { "dynamic" : "true", "_meta" : { }, "_source" : { "includes" : [], "excludes" : [] }, "dynamic_date_formats" : [ "strict_date_optional_time", "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z" ], "dynamic_templates" : [], "date_detection" : true, "numeric_detection" : false, "properties" : { "@timestamp" : { "type" : "date", "format" : "strict_date_optional_time" }, "@version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "host" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "instance_IP" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "instance_name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "java_class" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "level" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "log_message" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "message" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "path" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "rr" : { "type" : "text" }, "tags" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "tenant_id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "timestamp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "settings" : { "index" : { "creation_date" : "1615481578543", "number_of_shards" : "1", "number_of_replicas" : "1", "uuid" : "9o-UQnn-SKaj7LbhO8GYxQ", "version" : { "created" : "7070199" }, "provided_name" : "login.wso2.node.ip-2021.03.11" } } } } ``` What I need to do is to check if in the `message` field I have a SAML2 Response XML and if so I need to access to one value of this XML and count the unique occurrences. So far so good. The message field is multi mapping field. It is both text type and keyword type so I can use text type for full search and keyword type for aggregation, sorting and so on. What I did is to write this painless script: ```auto GET login.wso2.node.ip-2021.03.11/_search { "query": { "bool": { "filter": [ { "script": { "script": { "source": "doc['message.keyword'].value.contains('SAML_MESSAGES_LOGFILE') && doc['message.keyword'].value.contains('TINIT-')" } } } ] } }, "aggs": { "distinct_cf_count": { "scripted_metric": { "params": { "fieldName":"message" }, "init_script": "state.list = []", "map_script": """ //Controllo se c'è il campo message e se c'è fiscalnumber //if(doc[params.fieldName] != null && doc[params.fieldName].size()==0 ){ // def matcher = /(.*)<\/saml2:AttributeValue><\/saml2:Attribute>/.matcher(doc[params.fieldName].value); //if (matcher.find()) { // state.list.add(matcher.group(1)); //} if(doc[params.fieldName] != null && doc[params.fieldName].size()==0 && doc[params.fieldName].value.indexOf('TINIT-') > -1 ){ def valore = doc[params.fieldName].value; def startIdx = valore.indexOf('TINIT-')+'TINIT-'.length(); state.list.add(valore.substring(startIdx, 16)); } """, "combine_script": "return state.list;", "reduce_script": """ Map uniqueValueMap = new HashMap(); int count = 0; for(shardList in states) { if(shardList != null) { for(key in shardList) { if(!uniqueValueMap.containsKey(key)) { count +=1; uniqueValueMap.put(key, key); } } } } return count; """ } } } } ``` But I can't use regex because they are disabled and I should restart my ELK cluster in order to enable them. So I tried the `contains` and `indexOf` but I'm not able in counting the unique occorronces of this field. Do you have any suggestion? Thank you Angelo ---
**Author:** ![angeloimm](https://avatars.discourse-cdn.com/v4/letter/a/d2c977/32.png) [@angeloimm](https://discuss.elastic.co/u/angeloimm)\ **Post date:** [April 21, 2022, 2:00pm UTC](https://discuss.elastic.co/t/elk-painless-count-unique-distinct-occurrences/302582/2 "2022-04-21T14:00:27Z")
I gave a look. This check alwaus return 0 so it's like if message.keyword is always missing ```auto "map_script": """ //Controllo se c'è il campo message e se c'è fiscalnumber //if(doc[params.fieldName] != null && doc[params.fieldName].size()==0 ){ // def matcher = /(.*)<\/saml2:AttributeValue><\/saml2:Attribute>/.matcher(doc[params.fieldName].value); //if (matcher.find()) { // state.list.add(matcher.group(1)); //} **if( doc[params.fieldName].size()==0 ){** **state.list.add(UUID.randomUUID().toString());** **}** //else{ // def valore = doc[params.fieldName].value; // def cf = valore.splitOnToken('TINIT-')[1].substring(16); // state.list.add(cf); //} """, ``` Do you have any suggestion? I'm really blocked here... at 1 step to the solution ☹ ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 19, 2022, 2:01pm UTC](https://discuss.elastic.co/t/elk-painless-count-unique-distinct-occurrences/302582/3 "2022-05-19T14:01:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.