# How to use Elasticsearch to find Collocations and Statistically Improbable Phrases **URL:** \ **Category:** Elasticsearch\ **Created:** [July 6, 2013, 1:31am UTC](https://discuss.elastic.co/t/how-to-use-elasticsearch-to-find-collocations-and-statistically-improbable-phrases/12681 "2013-07-06T01:31:42Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Mike](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@Mike](https://discuss.elastic.co/u/Mike)\ **Post date:** [July 6, 2013, 1:31am UTC](https://discuss.elastic.co/t/how-to-use-elasticsearch-to-find-collocations-and-statistically-improbable-phrases/12681/1 "2013-07-06T01:31:42Z")
1. What is the best way to find collocations of terms? Is it by indexing documents using the shingles filter, and then performing a terms facet on that field? 2. How can I get a list of statistically improbable phrases for a document? From what I understand, instead of a list of highly frequent phrases like in the previous one, it would need to be a list based on the IDF score as well. Is there any way for me to extract that from elastic search? This is kind of the reverse of what a query returns, a list of phrases that would have a high score for a given document instead of a list of documents with a high score for a given phrase. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [July 6, 2013, 3:48am UTC](https://discuss.elastic.co/t/how-to-use-elasticsearch-to-find-collocations-and-statistically-improbable-phrases/12681/2 "2013-07-06T03:48:54Z")
Hi Mike, Are you looking to get this via ES because this looks like the best/easiest route or some other reason? Long time ago I ran a social bookmarking service where I took top N matches (with tags) and post-processed them. It worked OK with top 200 hits. I wouldn't shingle at index time. You'll shingle all your content and the index will balloon. Have a look at [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/products/key-phrase-extractor/index.html) - it's a Java lib you can use for getting Collocations and SIPs (and their hybrids). Unrelated to (Elastic)Search, but people sometimes use it in document processing pipelines to tag documents before indexing them (so you don't have to shingle everything in a doc, just add key terms/phrases as "tags"). You could then facet on that, or index in a separate highly weighted field, detect trending topics in a stream of data, etc. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Friday, July 5, 2013 9:31:42 PM UTC-4, Mike wrote: > 1. What is the best way to find collocations of terms? Is it by > indexing documents using the shingles filter, and then performing a terms > facet on that field? > > 2. How can I get a list of statistically improbable phrases for a > document? From what I understand, instead of a list of highly frequent > phrases like in the previous one, it would need to be a list based on the > IDF score as well. Is there any way for me to extract that from elastic > search? This is kind of the reverse of what a query returns, a list of > phrases that would have a high score for a given document instead of a list > of documents with a high score for a given phrase. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:27am UTC](https://discuss.elastic.co/t/how-to-use-elasticsearch-to-find-collocations-and-statistically-improbable-phrases/12681/3 "2017-07-06T02:27:52Z")