# Determine Top Keywords from the fetched list of documents **URL:** \ **Category:** Elasticsearch\ **Created:** [October 1, 2019, 2:12am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710 "2019-10-01T02:12:11Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Post date:** [October 1, 2019, 2:12am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/1 "2019-10-01T02:12:11Z")
This is a continuation of what I've posted months ago, but was not able to comment anymore because the post got locked. > [@Count number of keyword occurrences with query\_string](https://discuss.elastic.co/t/count-number-of-keyword-occurrences-with-query-string/190132): > > Hello and good day ElasticSearch Team and Developers! This is in relation with my StackOverflow post: [https://stackoverflow.com/questions/56932330/elasticsearch-count-number-of-keyword-occurences-in-a-document](https://stackoverflow.com/questions/56932330/elasticsearch-count-number-of-keyword-occurences-in-a-document) I'm searching through my index named web with a query\_string query as: ["query\_string" =\> ["default\_field" =\> $content, "query" =\> $keywords] ] Where as the $keywords variable contains: ("MCU" OR "Marvel" OR "Spiderman") Meaning, I'm looking for documents t… First, I would like to thank sir @Dadoonet for helping me out the last time, I was able to maximize the ES feature called **explain** , I just made a recursive function to fetch the specific "frequencies" array index from the ES response. Now this time my specific requirements are, I would like to determine which of my keywords would have the top number of occurrence. ```auto [ "query_string" => [ "default_field" => $content, "query" => $keywords ] ], ``` the `$keywords` variable contains: ```auto ("MCU" OR "Marvel" OR "Spiderman") ``` The problem with the **explain** feature is that it counts the frequency of these words occurred in **EACH AND EVERY document** but **NOT** from the **WHOLE LIST OF DOCUMENTS** received by the `search` query. **SOLUTIONS I'VE TRIED** **SOLUTION 1:** Since I'm using Laravel (a PHP based Framework), I've thought of looping throughout the `['hits']['hits'] ` part of the response. But I've thought of, what if I'll get 10,000 documents (the default maximum number of size)? That would be too much memory intensive for the server to loop throughout the `['hits']['hits']` and perform my recursive function just to sum all their frequencies. Is there an easier way for me to determine the top keywords from my list of fetched documents? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [October 1, 2019, 6:42am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/2 "2019-10-01T06:42:18Z")
Check out the ‘significant text’ aggregation. You can give it words of interest in the ‘include’ clause or let it discover the interesting words in your result set (the usual use case) For each word of interest it will give: - the “foreground” count (number of docs matching the query with the word) and - the background count (number of docs in the index with the word) ---
**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Post date:** [October 1, 2019, 7:21am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/3 "2019-10-01T07:21:57Z")
Awesome! Thank you so much for your response sir @Mark_Harwood! However, the response of the `significant_text` aggs also includes other "significant\_terms" as said by the response but for me, as the developer and for the client, they're not significant at all ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/5/35646430219e56541b97165b968499393fa2944a.png) And when I included the `$keywords`, it displays nothing ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/c/cc773c90cee6f2efdb4e35d55196127721b717f6.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/6/d6ad4ad04ee27654cfc80a5c6d1cf6785687f8bc.png) Is there any other option, sir? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [October 1, 2019, 7:42am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/4 "2019-10-01T07:42:42Z")
Can you share the query? (Preferably formatted JSON, not images) ---
**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Post date:** [October 1, 2019, 7:49am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/5 "2019-10-01T07:49:39Z")
I'm now using a different user together with his saved filters, please refer to the `aggs` portion sir {"index":"web","type":"index","body":{"query":{"bool":{"must":[{"query\_string":{"default\_field":"content","query":"(("Mayor Isko Moreno" OR "Mayor Vico Sotto") AND ("Manila" OR "Pasig"))"}},{"range":{"pub\_date":{"from":1569283200,"to":1569888000}}},{"bool":{"must\_not":[{"terms":{"mty\_id":[14,15,16]}}]}},{"bool":{"should":[{"bool":{"must\_not":[{"nested":{"path":"blk","query":{"bool":{"filter":{"match":{"blk.cli\_id":"4599"}}}}}},{"nested":{"path":"blk","query":{"bool":{"filter":{"match":{"blk.kgp\_id":1738}}}}}}]}}]}},{"bool":{"must\_not":[{"terms":{"pub\_id":["35209","35232","35268","35270","35296","35297","35298","35299","35300","35319","35374","35375","35376","35377","35382","35383","45208"]}}]}}]}},"aggs":{"KEYWORDS":{"significant\_text":{"field":"content","filter\_duplicate\_text":true,"include":"(("Mayor Isko Moreno" OR "Mayor Vico Sotto") AND ("Manila" OR "Pasig"))"}}},"sort":{"pub\_date":{"order":"desc"}}}} ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [October 1, 2019, 8:19am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/6 "2019-10-01T08:19:14Z")
Thanks for that. Your "include" clause will need fixing. It acts as a filter on the _indexed_ tokens found in the documents (so these will typically be lowercased words and possibly stemmed depending on your choice of analyzer). If you pass a string in the `include` clause it is interpreted as a regex. If you pass an array of strings they are interpreted as exact-match. It looks like you are passing a single, long, mixed case string which won't match any of the words. You may want to use the [analyze](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-analyze.html) api to convert your search terms like `Isco Moreno` into individual terms to use in this filter. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [October 1, 2019, 8:43am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/7 "2019-10-01T08:43:05Z")
> [@Noctis17](#): > > the response of the `significant_text` aggs also includes other "significant\_terms" as said by the response but for me, as the developer and for the client, they're not significant at all Many of these keywords look to have come from [the same press release](http://enjoyingwonderfulworld.blogspot.com/2019/09/gcash-powers-biggest-mobile-app-summit.html) The `filter_duplicate_text` setting is designed to help tune that sort of stuff out. Another significant text configuration that can help quality and speed of keyword discovery is to use a parent `sampler` aggregation to only look at a sample of high-quality matches. With large result sets it's also required to limit RAM use ---
**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Post date:** [October 1, 2019, 8:55am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/8 "2019-10-01T08:55:21Z")
Again, I would like to thank you for your heartfelt willingness to help, I've been really stuck about this problem for a quite a time now so do you mean sir my `$keywords` variable should look like an array of strings? like this? `["isko", "moreno"]` The problem with my `$keywords` variable is, they're actually a data from a **MySQL** database that are called **client\_keywords** and when I go to this certain app, I just collect these **client\_keywords** and concatenate them to become a **string** so I could search for them in the `query_string` query, is my understanding correct sir @Mark_Harwood? ---
**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Post date:** [October 1, 2019, 9:03am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/9 "2019-10-01T09:03:42Z")
and what's making this harder, our legacy app came from a Sphinxsearch full text server, we're just upgrading and migrating to Elasticsearch, so that's why we're transferring little by little, the **client\_keyword** of this client looks like this ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/6/46806424016dc9acf1d01cd12232ccaddad0cb97.png) It's based from the boolean search structure of Sphinx so that's why we have made a function that's converting these keywords into Elasticsearch readable boolean format, from the screenshot above, now to this: `(("Mayor Isko Moreno" OR "Mayor Vico Sotto") AND ("Manila" OR "Pasig"))` ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [October 1, 2019, 9:05am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/10 "2019-10-01T09:05:57Z")
> [@Noctis17](#): > > I just collect these **client\_keywords** and concatenate them to become a **string** so I could search for them in the `query_string` query, Pass them via the Analyze api like this: ``` GET /myindex/_analyze { "field": "content", "text":"My qUerY striNg" } ``` This will give you the response: ``` { "tokens" : [ { "token" : "my", "start_offset" : 0, "end_offset" : 2, "type" : "", "position" : 0 }, { "token" : "query", "start_offset" : 3, "end_offset" : 8, "type" : "", "position" : 1 }, { "token" : "string", "start_offset" : 9, "end_offset" : 15, "type" : "", "position" : 2 } ] } ``` You can take each of these tokens and use them in your aggregations. I've said before to look at `significant_text` aggregation but if you're not interested in background index stats or discovering new related terms then maybe the `adjacency_matrix` or `filters` aggregation will give you the information you need: ``` GET myindex/_search { "query": { "query_string": { "query": "My qUerY striNg" } }, "aggs": { "keywordUsage": { "adjacency_matrix": { "filters": { "my": { "term": { "content": "my" } }, "query": { "term": { "content": "query" } }, "string": { "term": { "content": "string" } } } } } } } ``` ---
**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Post date:** [October 1, 2019, 9:16am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/11 "2019-10-01T09:16:15Z")
This is sad, I think the `analyze` API isn't available for the PHP Api of Elasticsearch... ☹ I'm using Laravel (a PHP based framework) for the app sir ---
**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Post date:** [October 2, 2019, 2:00am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/12 "2019-10-02T02:00:28Z")
Up, is there a similar feature same as the `explain` but not on **document-level**? I mean, an `explain` feature that returns a detailed stats of the whole query instead of every document? I would like to thank sir Mark\_Harwood for helping me out, but the procedure he's introducing is quite tedious as the process from fetching the `$keywords` already take too much memory from the server ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [October 2, 2019, 7:08am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/13 "2019-10-02T07:08:48Z")
> [@Noctis17](#): > > take too much memory from the server The adjacency matrix approach in my last post shouldn’t use much memory? If you can’t use the analyze api to get the tokens I imagine splitting the search string on whitespace and using ‘match’ query rather than term query in the adjacency matrix filters would work fine. ---
**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Post date:** [October 2, 2019, 7:24am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/14 "2019-10-02T07:24:58Z")
Hello sir @Mark_Harwood, here's my problem I'm still trying to pursue the `significant_text` aggs ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/d/cd0617fc869da29c98e3471001f52c7761489463.png) The **1st** output is the list of my `$keywords` that are used for the `query_string` query The **2nd** output is my converted one, so from a string, I exploded them and placed on an array, and made them on lower case The **3rd** output shows a summary of my query Now, what's happening here is that, the `significant_text` can't look for my 1st and 2nd `$keywords` ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/7/17f6da3538c7a90f9d0d267c92a18035996ecadb.png) And P.S., I can't see the _foreground_ count that you have aforementioned in your replies above sir ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [October 2, 2019, 7:35am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/15 "2019-10-02T07:35:06Z")
> [@Noctis17](#): > > can't look for my 1st and 2nd `$keywords` They are phrases not words? Analysers typically tokenise input strings into words so these phrases won’t be in your index. > [@Noctis17](#): > > can't see the _foreground_ count that you have aforementioned in your replies That’s the ‘doc\_count’ ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [October 2, 2019, 7:38am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/16 "2019-10-02T07:38:21Z")
Reading your example though it looks to me like you want to understand the frequency of the different entities mentioned in the query, not single words. By that I mean the 2 mayors and the 2 locations. I’d steer you towards adjacency matrix again which can take arbitrary clauses as filters. ---
**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Post date:** [October 2, 2019, 7:40am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/17 "2019-10-02T07:40:09Z")
> [@Mark\_Harwood](#): > > They are phrases not words? Oh no if the `significant_text` only accepts words and not phrases, this would not be the ideal process for the users. Because the goal of our app is to let the users input `keywords` that they want to search from our database in a text-base scale.. any other way around sir? still testing the `adjacency matrix` ---
**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Post date:** [October 2, 2019, 7:45am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/18 "2019-10-02T07:45:31Z")
> [@Mark\_Harwood](#): > > Reading your example though it looks to me like you want to understand the frequency of the different entities mentioned in the query, not single words. > By that I mean the 2 mayors and the 2 locations. I’d steer you towards adjacency matrix again which can take arbitrary clauses as filters. yes yes! that's correct! you're understanding it right sir! just tested the `adjacency_matrix`, the bad thing about this is, it is counting the frequency based on **doc\_count** NOT frequency based on **word occurrence** I am looking for the same behavior as the `_explain` API, it counts the frequency of the keyword occurrence **PER DOCUMENT** , but my goal here is to determine the keyword occurrence all throughout the documents retrieved by my query ☹ ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [October 2, 2019, 8:14am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/19 "2019-10-02T08:14:48Z")
> [@Noctis17](#): > > it is counting the frequency based on **doc\_count** NOT frequency based on **word occurrence** Is that not typically the more useful measure? A single doc may be spammy and have a lot of repetition/keyword stuffing. What's the end objective? How does this information get used or benefit the end user? ---
**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\ **Post date:** [October 2, 2019, 8:29am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/20 "2019-10-02T08:29:49Z")
We will be displaying the Top Keywords given by the user for report purposes sir For example: The user entered: - Donald Trump - Border - Mexico Among these keywords, we'll be using the `query_string` query, to look up from the documents (articles) in our index and then display it to the user... Now, sooner or later the user will have a lot of **keywords** and we would like to give a report feature to display which among his/her keywords is the one appearing the most from the fetched articles [Next page](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710.md?page=2)