# Processor for shingles, similar to split?

**URL:** <https://discuss.elastic.co/t/processor-for-shingles-similar-to-split/266254>\
**Category:** Elasticsearch\
**Created:** [March 4, 2021, 2:46pm UTC](https://discuss.elastic.co/t/processor-for-shingles-similar-to-split/266254 "2021-03-04T14:46:26Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mark\_Petersen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_petersen/32/84916_2.png) [@Mark\_Petersen](https://discuss.elastic.co/u/Mark_Petersen)\
**Post date:** [March 4, 2021, 2:46pm UTC](https://discuss.elastic.co/t/processor-for-shingles-similar-to-split/266254/1 "2021-03-04T14:46:26Z")

</div>

Is there a processor that will do shingles or can I make a custom one somehow?

In the pipeline processor below, I split on the space character, but I'd also like to combine words like a shingle analyzer would:

```auto
PUT _ingest/pipeline/split
{
  "processors": [
    {
      "split": {
        "field": "title",
        "target_field": "title_suggest.input",
        "separator": "\\s+"
      }
    }
  ]
}

```

Example:

"Senior Business Developer" needs a suggestion field with these terms.

1. Senior Business Developer
2. Business Developer
3. Developer

Any ideas are appreciated, thanks!

---

<div class="post-metadata">

**Author:** ![Mark\_Petersen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_petersen/32/84916_2.png) [@Mark\_Petersen](https://discuss.elastic.co/u/Mark_Petersen)\
**Post date:** [March 5, 2021, 7:40pm UTC](https://discuss.elastic.co/t/processor-for-shingles-similar-to-split/266254/2 "2021-03-05T19:40:02Z")

</div>

Well I just created a script to do it. It's very basic but here it is:

```auto
PUT _ingest/pipeline/script
{
  "processors": [
    {
      "script": {
        "lang": "painless",
        "source": """
          if (!ctx.containsKey('title')) { return; }
          def title_words = ctx['title'].splitOnToken(' ');
          def title_suggest = [];
          for (def i = 0; i < title_words.length; i++) {
            def shingle = title_words[i];
            title_suggest.add(shingle);
            for (def j = i + 1; j < title_words.length; j++) {
              shingle = shingle + ' ' + title_words[j];
              title_suggest.add(shingle);
            }
            
          }
          ctx['title_suggest']=title_suggest;
        """
      }
    }
  ]
}

```

Usage:

```auto
PUT /item
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "suggest_analyzer": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": [
              "lowercase",
              "asciifolding"
            ]
          }
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text"
      },
      "title_suggest": {
        "type": "completion"
      }
    }
  }
}

PUT /item/_doc/1?pipeline=script
{
  "title": "Diabetes Mellitus Type 1"
}

```

Result:

```auto
GET /item/_doc/1

{
  "_index" : "item",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 24,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "title_suggest" : [
      "Diabetes",
      "Diabetes Mellitus",
      "Diabetes Mellitus Type",
      "Diabetes Mellitus Type 1",
      "Mellitus",
      "Mellitus Type",
      "Mellitus Type 1",
      "Type",
      "Type 1",
      "1"
    ],
    "title" : "Diabetes Mellitus Type 1"
  }
}

```

Note: It sucks that I can't just use the built-in shingle analyzer to break up the text into shingles and then insert that into another field.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 2, 2021, 7:40pm UTC](https://discuss.elastic.co/t/processor-for-shingles-similar-to-split/266254/3 "2021-04-02T19:40:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.