# URGENT: PLEASE HELP Unique list of bigram and trigram combinations then aggregate stats **URL:** \ **Category:** Elasticsearch\ **Created:** [June 28, 2018, 3:33pm UTC](https://discuss.elastic.co/t/urgent-please-help-unique-list-of-bigram-and-trigram-combinations-then-aggregate-stats/137817 "2018-06-28T15:33:25Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Lior\_Krolewicz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_krolewicz/32/32784_2.png) [@Lior\_Krolewicz](https://discuss.elastic.co/u/Lior_Krolewicz)\ **Post date:** [June 28, 2018, 3:33pm UTC](https://discuss.elastic.co/t/urgent-please-help-unique-list-of-bigram-and-trigram-combinations-then-aggregate-stats/137817/1 "2018-06-28T15:33:25Z")
I'm currently solving this poorly, not leveraging es. We recently scaled our data and my system crashes. I need an efficient solution to get back online. Thank you! I have a list of 300,000 unique strings with different number of words in each. Each string has statistics associated with it. I want to run a ngram, bigram, and trigram and aggregate the statistics by each unique combination (regardless of order) The order of the words does not matter. So if I have one line for "dog food for sale" with cost = $10 and another string for "for she cat food" with cost of $5, then i will have one item in my table for "sale food" and it will show up as $15. I want unique combinations then aggregate where string contains the single word (for ngram), contains both words in any order (bigram) and contains three words in any order. So like sum(cost) where word like "sale" (ngram). Sum(cost) where words like "sale" and words like "dog"... Etc The raw data is imported directly into es and the final output I want is one table where column one has all the unique combinations and colum two the sum of the cost per respective combination. What's the best way to do this? Thank you 😊 ---
**Author:** ![mayya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mayya/32/83147_2.png) [@mayya](https://discuss.elastic.co/u/mayya)\ **Post date:** [June 29, 2018, 12:42pm UTC](https://discuss.elastic.co/t/urgent-please-help-unique-list-of-bigram-and-trigram-combinations-then-aggregate-stats/137817/2 "2018-06-29T12:42:11Z")
It is not very clear to me what are your ES documents? Are you generating ngrams, bigrams and trigrams yourself and indexing them as documents? Or do you want ES extract them? If you want ES do this, ES has [Shingle Token Filter](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-shingle-tokenfilter.html) that can generate word ngrams, but the order is predefined. It is not very clear either what are your ES queries? What do you intend to search for? Or aggregate? ---
**Author:** ![Lior\_Krolewicz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_krolewicz/32/32784_2.png) [@Lior\_Krolewicz](https://discuss.elastic.co/u/Lior_Krolewicz)\ **Post date:** [July 3, 2018, 7:20pm UTC](https://discuss.elastic.co/t/urgent-please-help-unique-list-of-bigram-and-trigram-combinations-then-aggregate-stats/137817/3 "2018-07-03T19:20:19Z")
Thank you Maya, I want to break out all the words into ngram, bi gram and trigram, then all those unique items would be in one column Like WORD STAT dog food online 5 cat food online 5 TERM STAT food 10 online 10 dog 5 cat 5 dog food 5 cat food 5 food online 10 dog food online 5 cat food online 5 (I know it will generate more, but ill keep it simple) I dont care about the order when aggregating. So SUM(where word contains all strings in any order) Can we do all this in Elastic? Or do I need to do the ngram, bigram, and trigram in elastic, export, then aggregate.... this is what Im doing now, but when I try to put back the aggregated data into elastic it crashes. Is this possible? ---
**Author:** ![mayya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mayya/32/83147_2.png) [@mayya](https://discuss.elastic.co/u/mayya)\ **Post date:** [July 4, 2018, 8:46pm UTC](https://discuss.elastic.co/t/urgent-please-help-unique-list-of-bigram-and-trigram-combinations-then-aggregate-stats/137817/4 "2018-07-04T20:46:56Z")
I don't think Elasticsearch is designed for your task. What aggregation are you using? ---
**Author:** ![Lior\_Krolewicz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_krolewicz/32/32784_2.png) [@Lior\_Krolewicz](https://discuss.elastic.co/u/Lior_Krolewicz)\ **Post date:** [July 4, 2018, 10:05pm UTC](https://discuss.elastic.co/t/urgent-please-help-unique-list-of-bigram-and-trigram-combinations-then-aggregate-stats/137817/5 "2018-07-04T22:05:55Z")
Do you offer paid consulting? I'd love to share my screen and get a final answer 🙂 ---
**Author:** ![Lior\_Krolewicz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lior_krolewicz/32/32784_2.png) [@Lior\_Krolewicz](https://discuss.elastic.co/u/Lior_Krolewicz)\ **Post date:** [July 5, 2018, 8:42am UTC](https://discuss.elastic.co/t/urgent-please-help-unique-list-of-bigram-and-trigram-combinations-then-aggregate-stats/137817/6 "2018-07-05T08:42:49Z")
Do I'm looking to create a unique list of ngram - straight forward.. But then I need to generate a list of unique combinations of two words and three words. Because I don't care about the order, I don't think this is bigram and trigram. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 2, 2018, 8:42am UTC](https://discuss.elastic.co/t/urgent-please-help-unique-list-of-bigram-and-trigram-combinations-then-aggregate-stats/137817/7 "2018-08-02T08:42:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.