# Is it possible to eliminate duplication of search response when dealing with so long text?

**URL:** <https://discuss.elastic.co/t/is-it-possible-to-eliminate-duplication-of-search-response-when-dealing-with-so-long-text/182021>\
**Category:** Elasticsearch\
**Created:** [May 21, 2019, 2:15pm UTC](https://discuss.elastic.co/t/is-it-possible-to-eliminate-duplication-of-search-response-when-dealing-with-so-long-text/182021 "2019-05-21T14:15:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Masanori\_Ohnishi](https://avatars.discourse-cdn.com/v4/letter/m/278dde/32.png) [@Masanori\_Ohnishi](https://discuss.elastic.co/u/Masanori_Ohnishi)\
**Post date:** [May 21, 2019, 2:15pm UTC](https://discuss.elastic.co/t/is-it-possible-to-eliminate-duplication-of-search-response-when-dealing-with-so-long-text/182021/1 "2019-05-21T14:15:44Z")

</div>

## mapping

PUT /my\_index/\_mapping/blogpost

```auto
{
  "properties": {
    “document”: {
      "properties": {
        "content": { 
          "type": "text",
          "fields": {
            "raw": { 
              "type": "keyword",
              "ignore_above": 32766
            }
          }
        }
      }
    }
  }
}

```

## register

POST /my\_index/\_doc/1

```auto
{
  "contents": "so long text which exceeds 32766 bytes"
}

```

## search

GET /my\_index/\_search

```auto
{
    "query": {
        "match": {
            "contents": "somothing"
        }
    },
    "collapse" : {
        "field" : "contents.row" 
    }
}

```

Using this mapping, when searching text which is smaller than 32766 bytes,  
I can eliminate duplication of search response.

However, when searching text which is larger than 32766 bytes, I can't.

Is there another way of solving my requirements?

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [May 21, 2019, 2:30pm UTC](https://discuss.elastic.co/t/is-it-possible-to-eliminate-duplication-of-search-response-when-dealing-with-so-long-text/182021/2 "2019-05-21T14:30:14Z")

</div>

> [@Masanori\_Ohnishi](#):
>
> Is there another way of solving my requirements?

A hash of the content would be a shorter value to de-duplicate on but while it will have no false negatives (it will always recognise a duplicate text) it can have a small number of false positives (declaring non-duplicate texts as identical)

---

<div class="post-metadata">

**Author:** ![Masanori\_Ohnishi](https://avatars.discourse-cdn.com/v4/letter/m/278dde/32.png) [@Masanori\_Ohnishi](https://discuss.elastic.co/u/Masanori_Ohnishi)\
**Post date:** [May 28, 2019, 7:39am UTC](https://discuss.elastic.co/t/is-it-possible-to-eliminate-duplication-of-search-response-when-dealing-with-so-long-text/182021/3 "2019-05-28T07:39:04Z")

</div>

Thank you very much.

By using a hash of content, I could resolve the problem!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 25, 2019, 7:39am UTC](https://discuss.elastic.co/t/is-it-possible-to-eliminate-duplication-of-search-response-when-dealing-with-so-long-text/182021/4 "2019-06-25T07:39:05Z")

</div>

Closed on June 25, 2019, 7:39am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.