# Shingles which respect punctuation

**URL:** <https://discuss.elastic.co/t/shingles-which-respect-punctuation/2778>\
**Category:** Elasticsearch\
**Created:** [June 16, 2015, 10:28am UTC](https://discuss.elastic.co/t/shingles-which-respect-punctuation/2778 "2015-06-16T10:28:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mgvarley](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mgvarley/32/687_2.png) [@mgvarley](https://discuss.elastic.co/u/mgvarley)\
**Post date:** [June 16, 2015, 10:28am UTC](https://discuss.elastic.co/t/shingles-which-respect-punctuation/2778/1 "2015-06-16T10:28:00Z")

</div>

I am building an address matching engine for UK addresses in Elasticsearch and have found shingles to be very useful however I am seeing some issues when it comes to punctuation. A query for "4 Walmley Close" is returning the following matches:

1. Units 3 And 4, Walmley Chambers, 3 Walmley Close
2. Flat 4, Walmley Court, 10 Walmley Close
3. Co-Operative Retail Services Ltd, 4 Walmley Close

The true match is number 3, however both 1 and 2 match (falsely) as they both become '4 walmley' when turned into shingles. I would like to tell the shingle analyzer not generate shingles that straddle commas. So, for example 1) currently I get:

- units 3
- 3 and
- and 4
- 4 walmley
- walmley chambers
- chambers 3
- 3 walmley
- walmley close

...when in actual fact all I want is....

- units 3
- 3 and
- and 4
- walmley chambers
- 3 walmley
- walmley close

My current settings are below. I have experimented with swapping the tokenizer from standard to whitespace, this helps in that it retains the commas and would potentially avoid the situation above (i.e. I end up with '4, walmley' as my shingle in address 1 and 2) however I end up with lots of unusable shingles in my index and with 70 million documents I need to keep the index size down.

As you can see in the index settings I have also have a street\_sym filter which I would love to be able to use in my shingles e.g. for this example, in addition to generating 'walmley close' I would like to have 'walmley cl' however when I attempted to include this I got shingles of 'close cl' which were not terribly helpful!

Any advice from more experienced Elasticsearch users would be hugely appreciated. I have read through Gormley and Tong's excellent book but cannot get my head around this particular issue.

Thanks in advance for any help offered.

```
"analysis": {
    "filter": {
        "shingle": {
            "type": "shingle",
                "output_unigrams": false
        },
        "street_sym": {
            "type": "synonym",
                "synonyms": [
                "st => street",
                "rd => road",
                "ave => avenue",
                "ct => court",
                "ln => lane",
                "terr => terrace",
                "cir => circle",
                "hwy => highway",
                "pkwy => parkway",
                "cl => close",
                "blvd => boulevard",
                "dr => drive",
                "ste => suite",
                "wy => way",
                "tr => trail"
            ]
        }
    },
    "analyzer": {
        "shingle": {
            "type": "custom",
                "tokenizer": "standard",
                "filter": [
                "lowercase",
                "shingle"
            ]
        }
    }
}
```

---

<div class="post-metadata">

**Author:** ![Eduard\_Dudar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eduard_dudar/32/2845_2.png) [@Eduard\_Dudar](https://discuss.elastic.co/u/Eduard_Dudar)\
**Post date:** [March 16, 2016, 5:20am UTC](https://discuss.elastic.co/t/shingles-which-respect-punctuation/2778/2 "2016-03-16T05:20:18Z")

</div>

I'm looking for a solution to the similar problem. With raw Lucene access people split paragraphs manually but with ElasticSearch it's not quite possible.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:08pm UTC](https://discuss.elastic.co/t/shingles-which-respect-punctuation/2778/3 "2017-07-05T23:08:01Z")

</div>