# Determine Top Keywords from the fetched list of documents

**URL:** <https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710>\
**Category:** Elasticsearch\
**Created:** [October 1, 2019, 2:12am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710 "2019-10-01T02:12:11Z")\
**Posts on this page:** 4\
**Page:** 2

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [October 2, 2019, 8:53am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/21 "2019-10-02T08:53:55Z")

</div>

If you're echoing back the popularity of what users typed I'd argue that's of limited value.  
It's often more useful to give them things other than what they typed to open new lines of enquiry or refine the query.

Here's a real example of your query on news data using significant\_text

 ![Kibana-33](https://us1.discourse-cdn.com/elastic/original/3X/1/9/191095cced9680a9a64b7e0c417c2f6d8fce1167.png)

Note useful phrases like "border agents" are indexed because I used an Analyzer that produces single-word and two-word "shingles".

Another useful technique is to extract names into a structured field using something like rosette or spacy and then use the `annotated_text` field type to allow drill-downs into the text to show where they were mentioned. Here for example we use `significant_terms` on the `people` field (to focus on the significant rather than the popular) and discover the CBP commissioner:

 ![Kibana-34](https://us1.discourse-cdn.com/elastic/original/3X/4/d/4d03c009746f5b012f08f9304634785ed6fc5bf6.png)

These people names or significant text like "wall" or "barrier" can then be typically added to the query

---

<div class="post-metadata">

**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\
**Post date:** [October 3, 2019, 2:01am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/22 "2019-10-03T02:01:06Z")

</div>

> [@Mark\_Harwood](#):
>
> Is that not typically the more useful measure? A single doc may be spammy and have a lot of repetition/keyword stuffing.  
> What's the end objective? How does this information get used or benefit the end user?

Copy! Got it sir.

I'm getting more lenient to `doc_count` than the `keyword frequency` per document

My problem now is with the `adjacency_matrix` aggs, my `keywords` are on `string` data type. Since we're using Laravel (PHP-based framework) MySQL and Sphinx as the legacy database and search server, the `keywords` stored doesn't have similar patterns so I can't find a way to segregate the main keywords from the negated ones -- the ones in the **NOT** format of the `query_string` query (roughly we already have 25,000+ records now.

Going back to the `adjacency_matrix` aggs, here's my scenario:

I have this string as the user's keyword:

```
"(("Mayor Isko Moreno" OR "Mayor Vico Sotto") AND ("Manila" OR "Pasig"))"

```

With my attempt using the `significant_text` from our earlier replies, I have found a way to explode the string and convert it into an array

![image](https://us1.discourse-cdn.com/elastic/original/3X/a/5/a58c45dd2081e3c327511c4711c674dfbf0e9f32.png)

Now I have added it to the `adjacency_matrix` aggs

![image](https://us1.discourse-cdn.com/elastic/original/3X/4/8/4885a078e3800d32ff3976a9926270aeb30cbe2e.png)

And the result:

![image](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3d45f706c37f7d5e9841484d22576b59a1d0c7b0.png)

**The problem I'm seeing:**  
What I'm getting at, is that the `adjacency_matrix` looks like joining all the 4 _significant words_ from my `keywords` as one -- considering all documents that all 4 of them are appearing, but logically speaking, the `keyword` is only choosing one from `"Mayor Isko Moreno" OR "Mayor Vico Sotto"` **AND** `"Manila" OR "Pasig"`

This concept was able to be reproduced by the `explain` feature of Elasticsearch search API

_a sample screenshot of the `search` API response_  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ab5cad7877154526f81978728eb64fa1bc42d41.png)

_then here's a sample screenshot of the contents of the `explain` object_

 ![1](https://us1.discourse-cdn.com/elastic/original/3X/4/d/4d54c43126f857ed0cdf1d36f1a921224f5ea6b2.png)

And that's how we were able to display the keyword frequency PER HIT / DOCUMENT

The problem with `explain` is that it appears on **hits-level** --- meaning on every hit only  
 ![2](https://us1.discourse-cdn.com/elastic/original/3X/c/f/cf5e638278e30eb18efe8a52dd59d004b3c481dd.png)

Is there a way or a feature similar to `explain` that is appearing in the same level as the `body` so I can have a summarized result?

P.S. this is happening within just the `search` API, didn't need any other calls or endpoints

---

<div class="post-metadata">

**Author:** ![Noctis17](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noctis17/32/41656_2.png) [@Noctis17](https://discuss.elastic.co/u/Noctis17)\
**Post date:** [October 3, 2019, 7:09am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/23 "2019-10-03T07:09:25Z")

</div>

Latest R&D and update, my aggregation query now looks like this:

```
"aggs" => [
    "KEYWORDS" => [
        "filters" => [
            "filters" => [
                "term1" => [
                    "term" => [
                        'content' => "isko"
                    ]
                ],
                "term2" => [
                    "term" => [
                        'content' => "manila"
                    ]
                ]
            ]
        ]
    ]
]

```

and here's the response:

![image](https://us1.discourse-cdn.com/elastic/original/3X/3/5/352b408aa5c6180e41e9f1dd2bf98f4373f4c82e.png)

I'm almost near to my expected output, however when I wanted the **right** phrase / keywords, it is not being recognized by Elasticsearch

```
"aggs" => [
    "KEYWORDS" => [
        "filters" => [
            "filters" => [
                "term1" => [
                    "term" => [
                        'content' => "mayor isko moreno"
                    ]
                ],
                "term2" => [
                    "term" => [
                        'content' => "mayor vico sotto"
                    ]
                ]
            ]
        ]
    ]
]

```

_response_  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/e/0e111d39bf3bfb125620f76eecedccf8f520241a.png)

Does this mean Elasticsearch can't recognize phrases?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 31, 2019, 7:09am UTC](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710/24 "2019-10-31T07:09:29Z")

</div>

Closed on October 31, 2019, 7:09am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/determine-top-keywords-from-the-fetched-list-of-documents/201710.md?page=1)
