# How to make Kibana Terms panel to count complete string frequencies, rather than separate words frequencies? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 7, 2014, 1:34am UTC](https://discuss.elastic.co/t/how-to-make-kibana-terms-panel-to-count-complete-string-frequencies-rather-than-separate-words-frequencies/20099 "2014-10-07T01:34:34Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Konstantin\_Erman](https://avatars.discourse-cdn.com/v4/letter/k/9d8465/32.png) [@Konstantin\_Erman](https://discuss.elastic.co/u/Konstantin_Erman)\ **Post date:** [October 7, 2014, 1:34am UTC](https://discuss.elastic.co/t/how-to-make-kibana-terms-panel-to-count-complete-string-frequencies-rather-than-separate-words-frequencies/20099/1 "2014-10-07T01:34:34Z")
I have documents in ES with the field "Message", which normally represents some multi word text string. Trying to query it with Kibana to see which strings are in this property most frequently. What I actually get back is the table which shows frequency of the specific _words_, but not the whole strings! Now that I started to understand \*something \*about ES, my guess is that I supposed to map that "Message" field as { "type": "string", "index": "not\_analyzed" }, so it is not split into words. But on the other hand I still want to be able to find documents by searching for some words from their message fields. Next thought - multi\_field "mapping": { "type" : "string", "fields": { "raw": { "type": "string", "index": "not\_analyzed" } } } So that for normal query analysed Message field would work and when I build my Terms panel I use Message.raw instead. I need a confirmation that I'm moving in the right direction and this is optimal and intended way to achieve the goal. It does not look so elegant, that's why I'm asking. May be I miss some other ways to search string field using separate words, but still treat it as a whole for the purpose of counting. Please advise! Konstantin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/575eaefb-6be1-4a3a-b015-0051db56587f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/575eaefb-6be1-4a3a-b015-0051db56587f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Doug\_Nelson](https://avatars.discourse-cdn.com/v4/letter/d/c89c15/32.png) [@Doug\_Nelson](https://discuss.elastic.co/u/Doug_Nelson)\ **Post date:** [October 7, 2014, 3:50am UTC](https://discuss.elastic.co/t/how-to-make-kibana-terms-panel-to-count-complete-string-frequencies-rather-than-separate-words-frequencies/20099/2 "2014-10-07T03:50:22Z")
I use multi fields to have several different analysis types supported as need and also to have the raw version available like in your example. On Monday, October 6, 2014 8:34:34 PM UTC-5, Konstantin Erman wrote: > I have documents in ES with the field "Message", which normally represents > some multi word text string. Trying to query it with Kibana to see which > strings are in this property most frequently. What I actually get back is > the table which shows frequency of the specific _words_, but not the > whole strings! > > Now that I started to understand \*something \*about ES, my guess is that I > supposed to map that "Message" field as { "type": "string", "index": > "not\_analyzed" }, so it is not split into words. But on the other hand I > still want to be able to find documents by searching for some words from > their message fields. > > Next thought - multi\_field "mapping": > { > "type" : "string", > "fields": { > "raw": { "type": "string", "index": > "not\_analyzed" } > } > } > > So that for normal query analysed Message field would work and when I > build my Terms panel I use Message.raw instead. > > I need a confirmation that I'm moving in the right direction and this is > optimal and intended way to achieve the goal. It does not look so elegant, > that's why I'm asking. May be I miss some other ways to search string field > using separate words, but still treat it as a whole for the purpose of > counting. Please advise! > Konstantin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3641a396-0367-4cd3-8be5-6a1a110eefae%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3641a396-0367-4cd3-8be5-6a1a110eefae%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Konstantin\_Erman](https://avatars.discourse-cdn.com/v4/letter/k/9d8465/32.png) [@Konstantin\_Erman](https://discuss.elastic.co/u/Konstantin_Erman)\ **Post date:** [October 7, 2014, 4:59am UTC](https://discuss.elastic.co/t/how-to-make-kibana-terms-panel-to-count-complete-string-frequencies-rather-than-separate-words-frequencies/20099/3 "2014-10-07T04:59:06Z")
Does not it cause substantial inflation in the amount of data to be processed and stored at indexing time? As with most logs aggregation systems indexing is many orders of magnitude more frequent operation than querying and I'm concerned that using multi\_fields instead of all simple string fields may negatively impact indexing performance. May be there is a way to solve that problem at querying time? ALSO each field gets "primary" name and other names with the dot and the suffix for different representations. How to select which representation should be used as primary? On Monday, October 6, 2014 8:50:22 PM UTC-7, Doug Nelson wrote: > I use multi fields to have several different analysis types supported as need and also to have the raw version available like in your example. > > On Monday, October 6, 2014 8:34:34 PM UTC-5, Konstantin Erman wrote: > I have documents in ES with the field "Message", which normally represents some multi word text string. Trying to query it with Kibana to see which strings are in this property most frequently. What I actually get back is the table which shows frequency of the specific words, but not the whole strings! > > Now that I started to understand something about ES, my guess is that I supposed to map that "Message" field as { "type": "string", "index": "not\_analyzed" }, so it is not split into words. But on the other hand I still want to be able to find documents by searching for some words from their message fields. > > Next thought - multi\_field "mapping": > { > "type" : "string", > "fields": { > "raw": { "type": "string", "index": "not\_analyzed" } > } > } > > So that for normal query analysed Message field would work and when I build my Terms panel I use Message.raw instead. > > I need a confirmation that I'm moving in the right direction and this is optimal and intended way to achieve the goal. It does not look so elegant, that's why I'm asking. May be I miss some other ways to search string field using separate words, but still treat it as a whole for the purpose of counting. Please advise! > Konstantin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/510ce9e3-fd67-41c3-b969-b25e32eef352%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/510ce9e3-fd67-41c3-b969-b25e32eef352%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 7, 2014, 7:37am UTC](https://discuss.elastic.co/t/how-to-make-kibana-terms-panel-to-count-complete-string-frequencies-rather-than-separate-words-frequencies/20099/4 "2014-10-07T07:37:56Z")
Remember that Lucene is an inverted index and you can not deduce index volume from data input volume. If you have lots of millions of unique values = high cardinality, you will add this to Elasticsearch index volume, yes. If you have some thousand values = low cardinality in a field, you add close to nothing to index volume. You can even copy a field to many other fields and this adds close to nothing, this depends on the analyzer. There is no alternative to multi field: if you want to search analyzed forms, they must be in the index. If you want to examine field values from input for frequency, e.g. in aggregations, they must be in the index unchanged. Jörg On Tue, Oct 7, 2014 at 6:59 AM, Konstantin Erman [konste@gmail.com](mailto:konste@gmail.com) wrote: > Does not it cause substantial inflation in the amount of data to be > processed and stored at indexing time? > > As with most logs aggregation systems indexing is many orders of magnitude > more frequent operation than querying and I'm concerned that using > multi\_fields instead of all simple string fields may negatively impact > indexing performance. > > May be there is a way to solve that problem at querying time? > > ALSO each field gets "primary" name and other names with the dot and the > suffix for different representations. How to select which representation > should be used as primary? > > On Monday, October 6, 2014 8:50:22 PM UTC-7, Doug Nelson wrote: > > > I use multi fields to have several different analysis types supported as > > need and also to have the raw version available like in your example. > > > > On Monday, October 6, 2014 8:34:34 PM UTC-5, Konstantin Erman wrote: > > I have documents in ES with the field "Message", which normally > > represents some multi word text string. Trying to query it with Kibana to > > see which strings are in this property most frequently. What I actually get > > back is the table which shows frequency of the specific words, but not the > > whole strings! > > > > Now that I started to understand something about ES, my guess is that I > > supposed to map that "Message" field as { "type": "string", "index": > > "not\_analyzed" }, so it is not split into words. But on the other hand I > > still want to be able to find documents by searching for some words from > > their message fields. > > > > Next thought - multi\_field "mapping": > > { > > "type" : "string", > > "fields": { > > "raw": { "type": "string", "index": > > "not\_analyzed" } > > } > > } > > > > So that for normal query analysed Message field would work and when I > > build my Terms panel I use Message.raw instead. > > > > I need a confirmation that I'm moving in the right direction and this is > > optimal and intended way to achieve the goal. It does not look so elegant, > > that's why I'm asking. May be I miss some other ways to search string field > > using separate words, but still treat it as a whole for the purpose of > > counting. Please advise! > > Konstantin > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/510ce9e3-fd67-41c3-b969-b25e32eef352%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/510ce9e3-fd67-41c3-b969-b25e32eef352%40googlegroups.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGPpc%3DtvRO%3D93XM7X4rCQVkzP%2B\_dRhyJCFHWbpFvB9%2BWA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGPpc%3DtvRO%3D93XM7X4rCQVkzP%2B_dRhyJCFHWbpFvB9%2BWA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:57am UTC](https://discuss.elastic.co/t/how-to-make-kibana-terms-panel-to-count-complete-string-frequencies-rather-than-separate-words-frequencies/20099/5 "2017-07-06T00:57:59Z")