# ES can't find ❤️ love ❤️

**URL:** https://discuss.elastic.co/t/es-cant-find-love/43578
**Category:** Elasticsearch
**Created:** [March 5, 2016, 3:45pm UTC](https://discuss.elastic.co/t/es-cant-find-love/43578 "2016-03-05T15:45:01Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![Damien\_Alexandre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/damien_alexandre/32/21566_2.png) [@Damien\_Alexandre](https://discuss.elastic.co/u/Damien_Alexandre)
#### Post date: [March 5, 2016, 3:45pm UTC](https://discuss.elastic.co/t/es-cant-find-love/43578/1 "2016-03-05T15:45:01Z")

</div>

I'm playing with the strangest bug ever in ES. As I didn't clearly identified the root cause, I'm trying the forum first.

My analysis do:

- a char filter to replace `<3` by `❤`
- a synonym token filter to extend `❤` to `❤` and `love`.

If my content is `I ❤ ES`, my tokens are ok and I have a "love" token.

But if my content is `I <3 ES`, I only found `❤` in the tokens, not "love"!

Aren't the char\_filter supposed to be executed first?  
Also note that it does not happens with other replacements. If I replace all "ES" by "Elasticsearch" in the char\_filter, and then have synonyms for "Elasticsearch", they will be indexed.

Here is a "play" where I managed to reproduce the error: [https://found.no/play/gist/2493ccf2f307d46359a0#analysis](https://found.no/play/gist/2493ccf2f307d46359a0#analysis)

You can see that where there is a `<3`, no "love" synonym appears, but the replacement did occurs.

Thanks for your help!

---

<div class="post-metadata">

### Author: ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [March 5, 2016, 6:31pm UTC](https://discuss.elastic.co/t/es-cant-find-love/43578/2 "2016-03-05T18:31:36Z")

</div>

I can not reproduce this. In my version, there is no error.

> <https://gist.github.com/jprante/7503cb60e514d1ea46a4>

All 3 searches give 3 hits.

---

<div class="post-metadata">

### Author: ![Damien\_Alexandre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/damien_alexandre/32/21566_2.png) [@Damien\_Alexandre](https://discuss.elastic.co/u/Damien_Alexandre)
#### Post date: [March 7, 2016, 10:03am UTC](https://discuss.elastic.co/t/es-cant-find-love/43578/3 "2016-03-07T10:03:18Z")

</div>

Thanks for looking into it!

Yes you get the hit, that's because `<3` is translated to `❤`.

If you search only for `<3`, you can't find documents with `love`:

```
POST /test/doc/_search
{
    "query" : {
        "match" : {
            "text" : "<3"
        }
    }
}
# Only return <3 docs

```

That's because the synonyms are not applied to transformed emoticons ☹

You can see it, no "love" token here:

```
GET /test/_analyze?analyzer=myanalyzer
{
  "text": "I <3 ES"
} 

```

Tokens are "i", "❤" and "ES". Why not "love"! But it makes no sense to me, ❤ is supposed to pass in the synonyms token filter and be expended to "love".

❤

---

<div class="post-metadata">

### Author: ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [March 7, 2016, 4:29pm UTC](https://discuss.elastic.co/t/es-cant-find-love/43578/4 "2016-03-07T16:29:42Z")

</div>

I try to understand, you want to mix char\_filter and synonym filter?

This does not work, you are correct. I can only speculate about the reason. Maybe char\_filter has difficulties to pass tokens to synonym filter, or synonym filter can not pick up tokens.

Why not use synonym filter alone, without char\_filter, like this?

> <https://gist.github.com/jprante/10f6f25b9724c6c54a0d>

---

<div class="post-metadata">

### Author: ![Damien\_Alexandre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/damien_alexandre/32/21566_2.png) [@Damien\_Alexandre](https://discuss.elastic.co/u/Damien_Alexandre)
#### Post date: [March 7, 2016, 5:30pm UTC](https://discuss.elastic.co/t/es-cant-find-love/43578/5 "2016-03-07T17:30:14Z")

</div>

Glad you can see the issue too, I'm not crazy yet 😱 should I submit a ticket on github?

Plus, I can't use your solution because I have a filter removing punctuation from tokens **before** the synonym filter, allowing to handle emoji trapped in quotes, or snapped to a punctuation sign (The whitespace tokenizer produce a `❤.` token for the input `Es is ❤.`, the dot is an issue).

As `<3` and emoticons in general are mostly punctuation, I will be left with no match on the synonyms phase. That's why the char\_filter was supposed to be handy 🙂

---

<div class="post-metadata">

### Author: ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [March 7, 2016, 11:24pm UTC](https://discuss.elastic.co/t/es-cant-find-love/43578/6 "2016-03-07T23:24:14Z")

</div>

I'm not sure if you want just a demo solution for \<3, ❤, and 'love', it feels you want a general solution for all emoticons and what they mean, to get them translated into entities which can be searched.

If so, it is something that is far more than just fiddling with char\_filter and tokenizer.

Maybe you want emoticon segmentation and part-of-speech tagging. I know two solutions, but none of them is turnkey-ready for Elasticsearch:

1. Tweet Natural Language Processing - Part-of-speech Tagger [http://www.cs.cmu.edu/~ark/TweetNLP/#pos](http://www.cs.cmu.edu/~ark/TweetNLP/#pos)
2. Twitter's text common package with a Lucene-like Tokenizer at [https://github.com/twitter/commons/tree/master/src/java/com/twitter/common/text](https://github.com/twitter/commons/tree/master/src/java/com/twitter/common/text)

Alternative 1 is based on Java Pattern construction, while alternative 2 is more Lucene-focused and looks promising to get wrapped into an Elasticsearch tokenizer.

---

<div class="post-metadata">

### Author: ![Damien\_Alexandre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/damien_alexandre/32/21566_2.png) [@Damien\_Alexandre](https://discuss.elastic.co/u/Damien_Alexandre)
#### Post date: [March 8, 2016, 9:47pm UTC](https://discuss.elastic.co/t/es-cant-find-love/43578/7 "2016-03-08T21:47:12Z")

</div>

> [@jprante](#):
>
> you want a general solution for all emoticons and what they mean

I do, I'm building an emoji capable search engine, and came across this `<3 => ❤` bug when trying to deal with emoticons. I'm publishing my research and solutions pretty soon, and it works as expected except for ❤, which is why I created this thread.  
Seems to me there is a bug somewhere, I need to find some time to crack the case and maybe open an issue about it.

Thanks for the links to POS tokenizer, that's awesome 👍

---

<div class="post-metadata">

### Author: ![Damien\_Alexandre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/damien_alexandre/32/21566_2.png) [@Damien\_Alexandre](https://discuss.elastic.co/u/Damien_Alexandre)
#### Post date: [March 15, 2016, 1:32pm UTC](https://discuss.elastic.co/t/es-cant-find-love/43578/8 "2016-03-15T13:32:24Z")

</div>

Sorry for the double post,  
I wanted to show you where I got with this whole emoji search situation.

I've published an article about my research: [http://jolicode.com/blog/search-for-emoji-with-elasticsearch](http://jolicode.com/blog/search-for-emoji-with-elasticsearch)

The \<3 issue seems to have disappear with my latest tests, it may be thanks to the new clean-up I'm doing (see the char\_filters). Thanks again for the help you provided me.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:08pm UTC](https://discuss.elastic.co/t/es-cant-find-love/43578/9 "2017-07-05T23:08:19Z")

</div>


