# Extracting brands in documents using keyword and shingles

**URL:** https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873
**Category:** Elasticsearch
**Created:** [July 5, 2017, 10:02am UTC](https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873 "2017-07-05T10:02:29Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![fhelje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fhelje/32/19780_2.png) [@fhelje](https://discuss.elastic.co/u/fhelje)
#### Post date: [July 5, 2017, 10:02am UTC](https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873/1 "2017-07-05T10:02:29Z")

</div>

Hi,

I'm trying to use elastic to find brands in documents. So my idea was to create a index containing brands and search with a text to find brands.

So I tried this index and mapping:

```javascript
PUT brand
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer_keyword": {
          "type": "custom",
          "tokenizer": "keyword",
          "filter": [
            "asciifolding",
            "lowercase"
          ]
        },
        "my_analyzer_shingle": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "asciifolding",
            "lowercase",
            "shingle"
          ]
        }
      }
    }
  },
  "mappings": {
    "brand": {
      "properties": {
        "keyword": {
          "type": "text",
          "analyzer": "my_analyzer_keyword",
          "search_analyzer": "my_analyzer_shingle"
        }
      }
    }
  }
}

```

Some documents:

```javascript
POST /brand/brand/1
{
  "id": 1,
  "keyword": "nike"
}
POST /brand/brand/2
{
  "id": 2,
  "keyword": "adidas originals"
}

```

I then search like this:

```javascript
POST /brand/brand/_search
{
  "query": {
    "match": {
      "keyword": "I like nike shoes and adidas originals"
    }
  }
}

```

I expect to get **nike** and **adidas originals** as the result but I don't get anything back.  
I'm using elastic 5.4.3.

Is my thinking wrong?

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [July 5, 2017, 10:41am UTC](https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873/2 "2017-07-05T10:41:22Z")

</div>

Use the \_analyze API to understand what is happening at index time and query time.

You will see what tokens are exactly indexed and what terms are exactly compared to the inverted index at search time

---

<div class="post-metadata">

### Author: ![fhelje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fhelje/32/19780_2.png) [@fhelje](https://discuss.elastic.co/u/fhelje)
#### Post date: [July 5, 2017, 11:29am UTC](https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873/3 "2017-07-05T11:29:05Z")

</div>

Yes I have done that and from what I can see it should work but it doesn't. Here are my analysis.

Query:

```javascript
GET _analyze
{
   "tokenizer": "standard",
   "filter": [
      "asciifolding",
      "lowercase",
      "shingle"
   ],
   "char_filter": [
      "html_strip"
   ],
   "text": [
      "I like nike shoes and adidas originals"
   ]
}

```

Result (shortened for brevity):

```javascript
{
   "tokens": [
      ...
      {
         "token": "nike",
         "start_offset": 7,
         "end_offset": 11,
         "type": "<ALPHANUM>",
         "position": 2
      },
      {
         "token": "nike shoes",
         "start_offset": 7,
         "end_offset": 17,
         "type": "shingle",
         "position": 2,
         "positionLength": 2
      },
      ...
      {
         "token": "adidas",
         "start_offset": 22,
         "end_offset": 28,
         "type": "<ALPHANUM>",
         "position": 5
      },
      {
         "token": "adidas originals",
         "start_offset": 22,
         "end_offset": 38,
         "type": "shingle",
         "position": 5,
         "positionLength": 2
      },
      {
         "token": "originals",
         "start_offset": 29,
         "end_offset": 38,
         "type": "<ALPHANUM>",
         "position": 6
      }
   ]
}

```

Index:

```javascript
GET _analyze
{
   "tokenizer": "keyword",
   "filter": [
      "asciifolding",
      "lowercase"
   ],
   "char_filter": [
      "html_strip"
   ],
   "text": [
      "adidas originals"
   ]
}

```

Result:

```javascript
{
   "tokens": [
      {
         "token": "adidas originals",
         "start_offset": 0,
         "end_offset": 16,
         "type": "word",
         "position": 0
      }
   ]
}

```

---

<div class="post-metadata">

### Author: ![fhelje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fhelje/32/19780_2.png) [@fhelje](https://discuss.elastic.co/u/fhelje)
#### Post date: [July 5, 2017, 11:37am UTC](https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873/4 "2017-07-05T11:37:47Z")

</div>

Tried it on elastic version 2.4.4 and there it worked just fine.

Could it be a bug in latest version of elastic?

I got the expected result

```javascript
{
   "took": 73,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 2,
      "max_score": 0.003867892,
      "hits": [
         {
            "_index": "brand",
            "_type": "brand",
            "_id": "2",
            "_score": 0.003867892,
            "_source": {
               "id": 2,
               "keyword": "adidas originals"
            }
         },
         {
            "_index": "brand",
            "_type": "brand",
            "_id": "1",
            "_score": 0.003867892,
            "_source": {
               "id": 1,
               "keyword": "nike"
            }
         }
      ]
   }
}

```

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [July 5, 2017, 1:11pm UTC](https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873/5 "2017-07-05T13:11:21Z")

</div>

I agree that this is looking wrong.

Could you open an issue with this script?

```auto
DELETE brand
PUT brand
{
  "settings": {
    "number_of_shards": 1,
    "analysis": {
      "analyzer": {
        "my_analyzer_keyword": {
          "type": "custom",
          "tokenizer": "keyword",
          "filter": [
            "asciifolding",
            "lowercase"
          ]
        },
        "my_analyzer_shingle": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "asciifolding",
            "lowercase",
            "shingle"
          ]
        }
      }
    }
  },
  "mappings": {
    "brand": {
      "properties": {
        "keyword": {
          "type": "text",
          "analyzer": "my_analyzer_keyword",
          "search_analyzer": "my_analyzer_shingle"
        }
      }
    }
  }
}

POST /brand/brand
{
  "keyword": "nike"
}
POST /brand/brand
{
  "keyword": "adidas originals"
}
GET brand/_search
{
  "query": {
    "match": {
      "keyword": {
        "query": "I like nike shoes and adidas originals"
      }
    }
  }
}

```

---

<div class="post-metadata">

### Author: ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)
#### Post date: [July 5, 2017, 1:42pm UTC](https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873/6 "2017-07-05T13:42:40Z")

</div>

Unfortunately this is a side-effect of improvements that we made around handling query-time synonyms. The shingle filter generates a graph that confuses the query parser. See for instance the output of

```auto
GET brand/_validate/query?rewrite=true
{
  "query": {
    "match": {
      "keyword": {
        "query": "I like nike shoes and adidas originals"
      }
    }
  }
}

```

+1 to opening an issue

---

<div class="post-metadata">

### Author: ![fhelje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fhelje/32/19780_2.png) [@fhelje](https://discuss.elastic.co/u/fhelje)
#### Post date: [July 5, 2017, 2:21pm UTC](https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873/7 "2017-07-05T14:21:34Z")

</div>

Ok I'll open an Issue

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [July 5, 2017, 2:46pm UTC](https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873/10 "2017-07-05T14:46:34Z")

</div>

Awesome! And thanks for the detailed script. It helped a lot.

---

<div class="post-metadata">

### Author: ![fhelje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fhelje/32/19780_2.png) [@fhelje](https://discuss.elastic.co/u/fhelje)
#### Post date: [July 5, 2017, 3:21pm UTC](https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873/11 "2017-07-05T15:21:19Z")

</div>

Bug filed!

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 2, 2017, 3:21pm UTC](https://discuss.elastic.co/t/extracting-brands-in-documents-using-keyword-and-shingles/91873/12 "2017-08-02T15:21:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.