# Implement collocation in ElasticSearch

**URL:** <https://discuss.elastic.co/t/implement-collocation-in-elasticsearch/254118>\
**Category:** Elasticsearch\
**Created:** [November 3, 2020, 8:16am UTC](https://discuss.elastic.co/t/implement-collocation-in-elasticsearch/254118 "2020-11-03T08:16:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![a22xel](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@a22xel](https://discuss.elastic.co/u/a22xel)\
**Post date:** [November 3, 2020, 8:16am UTC](https://discuss.elastic.co/t/implement-collocation-in-elasticsearch/254118/1 "2020-11-03T08:16:39Z")

</div>

I am new in Elastic Search Usage I am trying to do an analyser or an ingest pipeline that would create collocations of words ( unigram, bigrams, trigrams with a step up to 2 ) . I am aware it's feasible in python but I am only interested in an ES solution.  
Up to know I tried to do it using shingles like this :

```auto
GET /_analyze
{
  "tokenizer": "standard",
  
  "filter": [
    {
      "type": "predicate_token_filter",
      "script": {
        "source": "token.getPosition() % 2 == 0"
      }
    },
    {
      "type": "shingle",
      "max_shingle_size": 5,
      "min_shingle_size": 3,
      "output_unigrams":false,
      "token_separator":" ",
      "filler_token":""
    },
    "trim",
    "unique",
    {
      "type":"pattern_replace",
      "pattern":"\\s+",
      "replacement":" "
    }
   
  ],
  "text": "aerial photo airplane taken with a nice camera"
}

```

it gives me this output  
output :

```auto
{
  "tokens" : [
    {
      "token" : "aerial airplane",
      "start_offset" : 0,
      "end_offset" : 21,
      "type" : "shingle",
      "position" : 0
    },
    {
      "token" : "aerial airplane with",
      "start_offset" : 0,
      "end_offset" : 32,
      "type" : "shingle",
      "position" : 0,
      "positionLength" : 3
    },
    {
      "token" : "airplane",
      "start_offset" : 13,
      "end_offset" : 28,
      "type" : "shingle",
      "position" : 1
    },
    {
      "token" : "airplane with",
      "start_offset" : 13,
      "end_offset" : 32,
      "type" : "shingle",
      "position" : 1,
      "positionLength" : 2
    },
    {
      "token" : "airplane with nice",
      "start_offset" : 13,
      "end_offset" : 39,
      "type" : "shingle",
      "position" : 1,
      "positionLength" : 3
    },
    {
      "token" : "with",
      "start_offset" : 28,
      "end_offset" : 35,
      "type" : "shingle",
      "position" : 2
    },
    {
      "token" : "with nice",
      "start_offset" : 28,
      "end_offset" : 39,
      "type" : "shingle",
      "position" : 2,
      "positionLength" : 2
    },
    {
      "token" : "nice",
      "start_offset" : 35,
      "end_offset" : 46,
      "type" : "shingle",
      "position" : 3
    }
  ]
}

```

but my ideal output would be ( just outputing the tokens)

```auto
['aerial', 'photo', 'airplane', 'taken', 'with', 'a', 'nice', 'camera', ('aerial', 'photo'), ('aerial', 'airplane'), ('aerial', 'taken'), ('photo', 'airplane'), ('photo', 'taken'), ('photo', 'with'), ('airplane', 'taken'), ('airplane', 'with'), ('airplane', 'a'), ('taken', 'with'), ('taken', 'a'), ('taken', 'nice'), ('with', 'a'), ('with', 'nice'), ('with', 'camera'), ('a', 'nice'), ('a', 'camera'), ('nice', 'camera'), ('aerial', 'photo', 'airplane'), ('aerial', 'photo', 'taken'), ('aerial', 'photo', 'with'), ('aerial', 'airplane', 'taken'), ('aerial', 'airplane', 'with'), ('aerial', 'taken', 'with'), ('photo', 'airplane', 'taken'), ('photo', 'airplane', 'with'), ('photo', 'airplane', 'a'), ('photo', 'taken', 'with'), ('photo', 'taken', 'a'), ('photo', 'with', 'a'), ('airplane', 'taken', 'with'), ('airplane', 'taken', 'a'), ('airplane', 'taken', 'nice'), ('airplane', 'with', 'a'), ('airplane', 'with', 'nice'), ('airplane', 'a', 'nice'), ('taken', 'with', 'a'), ('taken', 'with', 'nice'), ('taken', 'with', 'camera'), ('taken', 'a', 'nice'), ('taken', 'a', 'camera'), ('taken', 'nice', 'camera'), ('with', 'a', 'nice'), ('with', 'a', 'camera'), ('with', 'nice', 'camera'), ('a', 'nice', 'camera')]
```

---

<div class="post-metadata">

**Author:** ![mayya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mayya/32/83147_2.png) [@mayya](https://discuss.elastic.co/u/mayya)\
**Post date:** [November 9, 2020, 10:16pm UTC](https://discuss.elastic.co/t/implement-collocation-in-elasticsearch/254118/2 "2020-11-09T22:16:16Z")

</div>

An interesting problem.  
I think doing this through analysis in elasticsearch will not work, as the current shingle filters don't support `step`, and make shingles only with the adjacent words.  
There is a `multiplexer` filter that helps to produce parallel streams, but it doesn't work with shingles.  
You can create a github issue in the [elasticsearch repository](https://github.com/elastic/elasticsearch/issues) with a feature request for shingle filters to have this additional `step` parameter that allows to form shingles also with other words.

* * *

Finding collocations can be done through querying with [match\_phrase](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase.html) with a corresponding `slop` parameter. In this case, indexing just with `standard` analyzer should be sufficient.

But search only helps you to answer if terms in a phrase are collocated or not.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 7, 2020, 10:16pm UTC](https://discuss.elastic.co/t/implement-collocation-in-elasticsearch/254118/3 "2020-12-07T22:16:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.